FEASIBILITY  STUDY  OF  VARIANCE  REDUCTION 
IN  THE  LOGISTICS  COMPOSITE  MODEL 

THESIS 


George  P.  Cole,  III,  Captain,  USAF 
AFIT/GLM/ENS/07-04 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 


Wright-Patterson  Air  Force  Base,  Ohio 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED 


The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the 
official  policy  or  position  of  the  United  States  Air  Force,  Department  of  Defense,  or  the 
United  States  Government. 


AFIT/GLM/ENS/07-04 


FEASIBILITY  STUDY  OF  VARIANCE  REDUCTION 
IN  THE  COMPOSITE  MODEL 

THESIS 


Presented  to  the  Faculty 
Department  of  Operational  Sciences 
Graduate  School  of  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Air  University 

Air  Education  and  Training  Command 
In  Partial  Fulfillment  of  the  Requirements  for  the 
Degree  of  Master  of  Science  in  Logistics  Management 


George  P.  Cole,  III,  MS 
Captain,  USAF 

March  2007 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 


AFIT/GLM/ENS/07-04 


FEASIBILITY  STUDY  OF  VARIANCE  REDUCTION 
IN  THE  COMPOSITE  MODEL 


George  P.  Cole,  III,  MS 
Captain,  USAF 


Approved: 


Alan  W.  Johnson  (Advisor) 


date 


John  O.  Miller  (Reader) 


date 


AFIT/GLM/ENS/07-04 


Abstract 

The  Logistics  Composite  Model  (LCOM)  is  a  stochastic,  discrete-event 
simulation  that  relies  on  probabilities  and  random  number  generators  to  model  scenarios 
in  a  maintenance  unit  and  estimate  optimal  manpower  levels  through  an  iterative  process. 
Models  such  as  LCOM  involving  pseudo-random  numbers  inevitably  have  a  variance 
associated  with  the  output  of  the  model  for  each  run,  and  the  output  is  actually  a  range  of 
estimates.  The  reduction  of  the  variance  in  the  results  of  the  model  can  be  costly  in  the 
form  of  time  for  multiple  replications.  The  alternative  is  a  range  of  estimates  that  is  too 
wide  to  realistically  apply  to  real-world  maintenance  units. 

This  research  explores  the  application  of  three  different  methods  for  reducing  the 
variance  of  the  model’s  output.  The  methods  include  Common  Random  Numbers 
(CRN),  Control  Variates,  and  Antithetic  Variates.  The  differences  in  the  95%  confidence 
intervals  were  compared  between  the  variance  reduction  techniques  and  the  original 
model  to  determine  the  degree  of  variance  reduction.  The  result  is  a  successful  variance 
reduction  in  the  primary  output  statistics  of  interest  using  the  application  of  the  Control 
Variates  technique,  as  well  as  a  methodology  for  the  implementation  of  Control  Variates 
in  LCOM. 
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FEASIBILITY  STUDY  OF  VARIANCE  REDUCTION 


IN  THE  LOGISTICS  COMPOSITE  MODEL 


1.  Introduction 


1.1.  Background 

The  Logistics  Composite  Model  (LCOM)  is  one  of  the  Air  Force’s  primary  tools 
for  determining  optimal  logistics  and  maintenance  manpower  levels.  Additionally,  it  can 
be  used  to  model  other  logistics  resources  such  as  equipment  and  facilities.  The  LCOM 
is  a  stochastic,  discrete-event  simulation  that  relies  on  probabilities  and  random  number 
generators  to  model  scenarios  in  a  maintenance  unit  by  manipulating  certain  variables. 
Manpower  levels  are  attained  through  an  iterative  process  in  which  the  variables 
consisting  of  supply,  facilities,  and  equipment  are  set  based  on  command  standards. 
Manpower  levels  are  adjusted  after  each  run  until  a  desired  Sortie  Generation  Rate  (SGR) 
is  attained  (Boyle  1990). 

In  a  model  such  as  LCOM,  many  real-life  characteristics  exhibit  random 
behavior.  As  Law  and  Kelton  (2004)  state,  “A  simulation  of  any  system  or  process  in 
which  there  are  inherently  random  components  requires  a  method  of  generating  or 
obtaining  numbers  that  are  random ,  in  some  sense”  (Law  and  Kelton  2004).  The  random 
number  generators  aid  the  customer  in  simulating  the  randomness  of  the  system  by 
producing  a  stream  of  continuous,  uniformly  distributed  numbers  between  0  and  1 .  The 
intent  of  the  random  number  generator  is  to  produce  these  numbers  independently. 
However,  the  computer  is  actually  using  a  recursive  algorithm  that  produces  numbers  that 
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seem  independent,  but  instead  follow  a  pattern  that  can  be  repeated  over  and  over,  called 
a  stream  (Kelton  et  al.  2004).  These  types  of  random  number  generators  are  called 
pseudo-random  generators. 

Models  involving  pseudo-random  numbers  inevitably  have  a  variance  associated 
with  the  output  of  the  model  for  each  run,  and  the  output  is  actually  a  range  of  estimates. 
The  reduction  of  the  variance  in  the  results  of  the  model  can  be  costly  in  the  form  of  time 
for  multiple  replications  or  producing  a  range  of  estimates  that  is  too  wide  to  realistically 
analyze.  Since  simply  increasing  the  number  of  replications  is  not  always  realistic  for 
reducing  variance,  this  paper  proposes  the  application  of  other  methods  for  reducing  the 
variance  of  the  model’s  output.  The  methods  include  Common  Random  Numbers 
(CRN),  Control  Variates  (CV),  and  Antithetic  Variates  (AV).  When  applied  to  a  model 
such  as  LCOM,  these  variance  reduction  techniques  may  significantly  reduce  the 
variance  without  increasing  the  number  of  replications  (Law  and  Kelton  2000). 

1.2.  Problem  Statement 

Due  to  the  complexity  of  the  LCOM’s  main  model  in  simulating  a  real-life 
maintenance  unit,  the  model  contains  many  instances  involving  pseudo-random  number 
generation.  The  large  amount  of  randomness  in  the  model  causes  results  that  display  a 
significant  amount  of  variance.  As  with  any  other  model,  large  variances  in  the  output 
are  unfavorable.  The  LCOM  office  in  the  Aeronautical  Systems  Center’s  (ASC)  Systems 
Supportability  Analysis  Branch,  Wright-Patterson  AFB,  Ohio,  is  interested  in  finding 
ways  to  reduce  the  variance  in  the  model’s  output. 
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1.3.  Research  Objective 

The  objective  of  this  research  is  to  replicate  prior  efforts  by  Bednar  (2005)  by 
studying  the  impact  of  various  variance  reduction  techniques  on  LCOM.  The  research 
will  apply  these  techniques  to  the  model  and  analyze  the  results,  determine  whether  or 
not  each  technique  is  effective  and  identify  the  most  effective  method. 

1.4.  Research  Focus 

This  research  first  examines  the  capabilities  and  effectiveness  of  the  model’s 
random  number  generators.  We  will  then  use  this  information  to  investigate  the 
application  of  several  variance  reduction  techniques.  In  particular,  three  classic 
techniques  -  common  random  numbers,  antithetic  variates,  and  control  variates  -  are 
applied  to  the  LCOM  model  with  the  intention  of  reducing  the  variance  of  the  model’s 
output. 

1.5.  Methodology 

Before  we  can  apply  these  techniques,  we  must  examine  the  random  number 
generator  used  in  the  LCOM  and  determine  if  the  generator  is  suitably  robust  for  the 
application  of  the  variance  reduction  methods.  If  not,  the  random  number  generator  must 
be  replaced  with  a  more  robust  generator  capable  of  facilitating  synchronization  and 
numerous  replications. 

The  common  random  numbers  approach  involves  multiple  scenarios  in  the  same 
model.  Using  this  approach,  the  individual  sources  of  randomness,  or  random  variates, 
are  synchronized  using  the  same  random  number  stream  across  the  two  scenarios.  Then, 
configurations  in  two  different  scenarios  will  use  the  same  random  numbers  so  that  the 
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different  scenarios  can  experience  similar  experimental  conditions  (Law  and  Kelton 

2000). 

The  second  technique,  control  variates,  involves  identifying  potential  control 
variates  within  the  model  that  can  be  used  to  reduce  the  variance  in  the  output.  This 
method  requires  the  identification  of  a  particular  random  variable  or  variables  with 
known  expected  values  that  are  thought  to  correlate  to  an  output  variable,  either 
positively  or  negatively.  Then,  using  these  potential  control  variates  and  the  estimated 
correlation,  the  expected  value  of  the  output  variable  is  adjusted  up  or  down  based  on  the 
differences  between  the  observed  values  of  the  control  variates  and  their  known  expected 
values  (Kelton  et  al.  2004). 

The  last  technique,  antithetic  variates,  “attempts  to  induce  negative  correlation 
between  the  results  of  one  replication  and  another,  and  uses  this  correlation  to  reduce 
variance”  (Kelton  et  al.  2004).  This  involves  a  second  replication  that  replaces  each 
random  number  U,  with  the  random  number  1  - U,.  For  example,  where  U,  is  used  for  a 
particular  purpose,  1-L,is  used  in  the  second  replication  for  the  same  purpose.  The  pairs 
are  averaged,  possibly  replicating  this  for  several  pairs  (Law  and  Kelton  2000). 

The  approach  for  the  comparison  of  these  three  methods  involves  acquiring  the 
source  code  for  the  LCOM  model  and  manipulating  the  calls,  or  random  number 
generators,  to  perfonn  each  particular  method  and  comparing  the  output  variances  of 
each  method  as  well  as  the  output  variance  of  the  original  model. 
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1.6.  Assumptions/Limitations 

This  research  assumes  the  LCOM’s  source  code  can  be  manipulated  and  re¬ 
compiled  to  allow  the  application  of  the  three  variance  reduction  techniques  -  common 
random  numbers,  antithetic  variates,  and  control  variates. 

1.7.  Implications 

This  research  has  several  implications  for  positive  impact.  First,  the  variance 
reduction  techniques  could  potentially  benefit  the  LCOM  users  by  requiring  fewer 
replications  and  shorter  process  time  to  achieve  a  given  confidence  in  a  model’s  output. 
This  could  allow  the  users  to  save  processing  time  and  test  additional  scenarios  instead  of 
performing  additional  replications.  Second,  the  ASC  LCOM  office  has  developed  an 
optimizer,  called  an  auto-constraining  wrapper,  which  automatically  perfonns  the 
constraining  process  of  defining  resource  levels  (Boughton  2006).  These  variance 
reduction  techniques  could  potentially  improve  the  perfonnance  of  the  optimizer  and 
enhance  the  overall  effectiveness  of  the  LCOM.  In  the  long  term,  this  could  lead  to  more 
precise  manpower  levels  to  meet  mission  requirements  and  more  accurate  sortie 
generation  rate  forecasts. 

1.8.  Overview 

There  are  five  chapters  in  this  research.  Chapter  1,  the  Introduction,  contains 
background  information,  problem  statement,  research  objective,  a  synopsis  of  the 
methodology,  and  the  assumptions  of  the  research.  Chapter  2,  the  Literature  Review, 
contains  descriptions  of  LCOM,  other  LCOM  research,  random  number  generators, 
common  random  numbers,  antithetic  variates,  and  control  variates.  Chapter  3,  the 
Methodology,  discusses  how  the  variance  reduction  techniques  are  applied  to  LCOM. 
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Chapter  4,  Results  and  Analysis,  presents  the  output  of  the  results  from  the  application  of 
the  three  techniques.  These  results  are  analyzed  to  determine  the  effectiveness  of  each 
technique.  Chapter  5,  Conclusions,  provides  a  discussion  of  the  analysis  and  results  in 
the  previous  chapter  as  well  as  recommendations  for  future  research  in  the  application  of 
variance  reduction  techniques  and  LCOM. 
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2.  Literature  Review 


2.1.  LCOM 

The  U.S.  Air  Force’s  Logistics  Composite  Model  has  existed  since  the  late  1960s, 
created  through  a  combined  effort  by  the  Rand  Corporation  and  the  Air  Force  Logistics 
Command  to  “relate  base-level  logistics  resources  with  each  other  and  with  sortie 
generating  capability”  (Boyle  1990).  While  the  model  is  capable  of  studying  the 
interactions  between  several  variables,  it  has  evolved  to  be  known  as  one  of  the  Air 
Force’s  primary  tools  for  establishing  manpower  levels  in  operational  maintenance  units 
and  exists  as  part  of  the  Air  Force’s  Standard  Analysis  Toolkit  (AFSAT)  (Dierker  2006). 

Two  separate  versions  of  the  LCOM  exist  in  the  Air  Force  today,  one  at  the  Air 
Force  Manpower  Agency  (AFMA),  Randolph  AFB,  Texas,  and  the  other  at  the 
Aeronautical  Systems  Center’s  (ASC)  Systems  Supportability  Analysis  Branch,  Wright- 
Patterson  AFB,  Ohio  (Pettingill  2003).  These  two  separate  models  essentially  perform 
the  same  function,  with  some  minor  differences  in  the  user  interface  (Dierker  2006).  The 
AFMA  version  has  four  primary  users:  Air  Combat  Command,  Air  Mobility  Command, 
Air  Force  Special  Operations  Command,  and  Air  Education  and  Training  Command. 

The  ASC  is  the  only  primary  user  for  the  ASC  version  (Dawson  2006).  The  AFMA 
version  is  used  by  the  MAJCOMs  to  derive  65-70%  of  their  maintenance  manpower 
requirements  (Dawson  2006).  The  rest  comes  from  Air  Force  Instructions  and  other 
guidance.  The  ASC  version  is  used  “to  analyze  manpower  requirements  for  acquired 
weapon  systems  (as  well  as  evaluate  manpower  requirement  changes  resulting  from 
modifications  to  current  weapon  systems)”  (Dawson  2006). 
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The  LCOM  model  consists  of  multiple  submodels,  including  an  input  model,  a 
main  model,  and  several  post  processors.  The  input  model  analyzes  input  data  from  the 
user  and  makes  assumptions  and  corrections  when  necessary  so  that  the  data  can  be  used 
by  the  main  model  and  post  processors.  This  data  typically  includes  maintenance  data 
from  the  Air  Force’s  Maintenance  Data  Collection  systems,  essential  tasks  needed  to  be 
performed  to  service  each  aircraft,  mission  requirements  and  flying  times.  The  main 
model  is  the  heart  of  the  simulation,  and  the  primary  source  of  data  for  our  research.  It 
uses  maintenance  data  and  sortie  data  together  with  the  process  logic  shown  in  Figure 
2.1. 


Figure  2.1:  LCOM  Simulation  Logic  (ASC/ENM  2004) 

Various  post  processors  show  simulation  results  as  a  function  of  time,  such  as  manpower 
demands,  resource  and  facility  usage,  parts  availability,  and  depot  workload  (ASC/ENM 
2004). 

The  LCOM  is  subject  to  four  forms  of  variance:  stochastic  model  variance, 
interviewee  variance,  analyst  variance,  and  MAJCOM  procedure  variance  (Dawson 
2006).  Stochastic  model  variance  occurs  because  the  random  number  generators  generate 
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failures  and  repairs  randomly,  resulting  in  a  random  output.  The  random  output  is  the 
type  of  variance  this  research  examines.  The  other  fonns  of  variance  are  associated  with 
data  collection,  evaluation  and  interpretation  of  results  by  analysts,  and  differing 
processes  and  philosophies  across  MAJCOMs  (Dawson  2006). 

2.2.  Previous  LCOM  Studies 

Because  of  its  longevity,  complexity,  and  broad  application,  the  LCOM  has  been 
the  subject  of  many  studies.  Most  studies  have  been  focused  on  either  the  applicability  of 
LCOM  to  various  levels  and  resources  of  the  Air  Force,  or  the  comparing  LCOM  to  other 
similar  models.  Below  is  a  brief  summary  of  the  most  notable  research  efforts  involving 
the  inner  workings  of  LCOM. 

In  2006,  the  Air  Force  Logistics  Management  Agency  performed  a  study  that 
examined  LCOM  process  reengineering.  The  study  examined  the  LCOM  development 
process  and  the  steps  required  for  developing  and  conducting  an  LCOM  study  for  a 
particular  MAJCOM.  The  report  recommended  seven  potential  changes  aimed  at 
reducing  the  overall  development  time  (Dawson  2006). 

Captain  Kirk  B.  Pettingill  performed  a  study  in  2003  on  the  LCOM’s  ability  to 
determine  a  maintenance  unit’s  current  capacity,  rather  than  just  a  front-end  tool  for 
setting  initial  manpower  levels.  He  collected  actual  data  from  three  flying  units  and 
compared  the  data  to  LCOM  outputs.  He  concluded  that  the  LCOM  is  an  effective  tool 
for  detennining  a  maintenance  unit’s  current  capacity  to  produce  sorties,  but  with 
limitations  (Pettingill  2003). 

In  1996,  Captain  Todd  Carrico  and  Patrick  K.  Clark  of  the  Human  Resources 
Directorate,  Logistics  Research  Division  performed  research  on  the  automated 
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conversion  of  LCOM  into  Integrated  Model  Development  Environment  (IMDE)  objects. 
IMDE  is  a  simulation  development  system  that  embeds  an  object-oriented  modeling 
approach  within  an  interface  to  improve  the  user-friendliness  of  the  LCOM.  This 
research  helped  update  the  LCOM  model  to  a  more  graphical  interface  orientation  for  the 
user  (Carrico  and  Clark  1996). 

In  1981,  a  research  effort  performed  by  Robert  Garcia  and  Joseph  P.  Racher,  Jr. 
focused  on  the  variance  in  workcenter  performance  based  on  skill  level  mixtures.  They 
measured  skill  level  effects  and  detennined  that  skill  mixture  has  an  impact  on 
workcenter  perfonnance.  Garcia  and  Racher  developed  a  methodology  for  capturing  this 
effect  and  incorporating  it  into  the  LCOM  model  (Garcia  and  Racher  1981). 

2.3.  SIMSCRIPT  1 1.5 

The  LCOM  model  is  programmed  using  the  California  Analysis  Center,  Inc’s 
(CACI)  simulation  language  called  SIMSCRIPT  II. 5.  The  software  is  relatively  easy  to 
use,  with  interactive  graphical  user  interfaces  and  animated  graphics  (CACI  2006).  The 
following  excerpt  is  from  CACI’s  manual  titled,  Building  Simulation  Models  with 
SIMSCRIPT  II.  5: 

SIMSCRIPT  II. 5  is  an  integrated,  interactive  development  environment 
controlled  by  SimLab.  SimLab  includes  the  complete  SIMSCRIPT  II. 5 
programming  language,  utilities  for  editing  and  managing  SIMSCRIPT 
II. 5  programs,  the  SIMGRAPHICS  I  and  II  graphical  interface  and 
utilities,  and  comprehensive  online -help  (Russell  1999). 

Typical  SIMSCRIPT  II. 5  applications  include  telecommunications,  network  analysis, 

transportation,  manufacturing,  health  care,  and  military  operations  to  include  wargaming 

and  logistics  planning  (CACI  2006). 
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The  current  compiling  environment  for  SIMSCRIPT  II. 5  has  evolved  from 
SimLab  and  is  called  Simulation  Studio,  or  SimStudio  (CACI  2006).  It  provides  support 
for  projects  with  hierarchical  directories,  and  is  available  on  all  supported  SIMSCRIPT 
II. 5  platforms  including  Windows,  PC  Linux,  and  UNIX  workstations  (CACI  2006). 
SimStudio  “...has  a  more  intuitive  graphical  user  interface,  a  modem  look-and-feel,  and 
incorporates  SIMSCRIPT  II. 5  Syntax  Color  Coded  Text  Editor  and  all  Graphical  Editors 
for  SIMSCRIPT  II.5  Graphics”  (CACI  2006). 

2.4.  Variance  Reduction  Techniques 

Since  models  with  random  input,  such  as  LCOM,  consequently  produce  random 
output,  there  is  a  variance  associated  with  the  output  over  a  given  number  of  replications 
(Law  and  Kelton  2000).  For  the  purpose  of  analysis  and  interpretation,  the  amount  of 
computational  time  and  appropriate  statistical  analysis  can  often  be  great.  As  Law  and 
Kelton  (2000)  state,  “Sometimes  the  cost  of  even  a  modest  statistical  analysis  of  the 
output  can  be  so  high  that  the  precision  of  the  results,  perhaps  measured  by  confidence- 
interval  width,  will  be  unacceptably  poor,”  so  therefore  the  analyst  should  “. .  .try  to  use 
any  means  possible  to  increase  the  simulation’s  efficiency.”  By  the  term  efficiency  they 
are  talking  about  statistical  efficiency,  measured  by  the  variances  of  the  output  random 
variables  from  the  simulation  (Law  and  Kelton  2000).  Certain  variance  reduction 
techniques,  when  properly  applied,  may  result  in  greater  precision  over  fewer  replications 
without  disturbing  its  expectation  (Law  and  Kelton  2000).  This  research  focuses  on  three 
of  these  techniques:  common  random  numbers  (CRN),  antithetic  variates  (AV),  and 
control  variates  (CV).  These  techniques  are  described  in  depth  below. 
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2.4.1  Common  Random  Numbers 


The  common  random  numbers  technique  is  unlike  the  other  two  techniques  in  that 
it  is  a  multiple  model  technique,  while  CV  and  AV  are  both  single  model  techniques. 

This  means  that  CRN  involves  the  comparison  of  two  or  more  system  configurations 
instead  of  investigating  a  single  configuration  (Law  and  Kelton  2000).  CRN  requires  that 
the  different  configurations  not  only  use  the  same  random  numbers,  but  that  the  numbers 
are  synchronized  to  induce  similar  experimental  conditions.  This  technique,  also  known 
as  matched  streams,  or  matched  pairs,  ensures  that  “. .  .any  observed  differences  in 
performance  are  due  to  differences  in  the  system  configurations  rather  than  to 
fluctuations  of  the  ‘experimental  conditions’  ”  (Law  and  Kelton  2000).  The  following  is 
a  summary  of  the  common  random  numbers  theory  described  in  Law  and  Kelton’ s 
Simulation  Modeling  and  Analysis  (2000): 

Consider  the  case  of  two  alternative  configurations,  where  Xy  and  Xy  are  the 
observations  from  the  first  and  second  configurations  on  the  /th  independent  replication, 
and  we  want  to  estimate  equation  (2.1): 

(2.1) 

For  n  replications  where  j  =  1,2,..  .n,  and  E[Zy]  =  L  ,  and  Zj  =  Xy  -  Xy,  then  equation  (2.2) 
is  an  unbiased  estimator  of 

2  =  {tz,  (2.2) 

’l  j  =  1 


Furthermore, 
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(2.3) 


Var(Zy)  _  Var(xii.)  + Var(x2  .)-2Cov(xiy,X27) 
n  n 

Obviously,  if  the  simulations  of  the  two  different  configurations  are  done  independently, 
then 

Cov(x,pX2J)=0  (2.4) 

However,  if  there  was  a  way  to  induce  a  positive  correlation,  then 

Cov(z,.,X2.)>0  (2.5) 

and  the  value  for  Var  (Z )  will  be  reduced  (Law  and  Kelton  2000). 

The  key  to  the  induction  of  this  positive  correlation  is  the  synchronization  of 
random  variate  draws  across  the  different  system  configurations  on  the  same  replication. 
As  Law  and  Kelton  (2000)  state,  “ Ideally ,  a  specific  random  number  used  for  a  specific 
purpose  in  one  configuration  is  used  for  exactly  the  same  purpose  in  all  other 
configurations.”  The  process  of  synchronization  involves  two  steps.  First,  all  points  in 
the  model  where  a  random  number  or  variate  is  drawn  must  be  identified.  Second,  each 
point  is  assigned  its  own  random  number  stream  (Bednar  2005).  Care  should  also  be 
taken  to  ensure  that  streams  of  random  numbers  do  not  overlap.  For  this  reason,  a  robust 
random  number  generator  with  built-in  functions  that  keep  track  of  random  number 
streams  should  be  used  (Bednar  2005). 

2.4.2  Antithetic  Variates 

The  antithetic  variates  technique,  like  CRN,  induces  a  correlation  between 
separate  runs.  However,  AV  is  conducted  on  a  single  configuration.  Also,  the  desired 


Var(z) 
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correlation  is  negative  rather  than  positive  (Law  and  Kelton  2000).  AV  tries  to  induce 
this  negative  correlation  by  making  pairs  of  runs  of  the  model  where  complementary 
random  variates  drive  the  two  runs  in  a  pair.  The  complementary  random  variates  are 
such  that  if  Uj  is  a  particular  random  number  used  for  a  particular  purpose  in  the  first  run 
j,  then  1  —  L7,  is  used  for  the  same  purpose  in  the  second  run.  The  use  of  I  -  Uj  is  valid 
since  U  ~  t/(0,l)  then  [1  -  U]  is  also  ~  L/(0, 1 )  (Law  and  Kelton  2000).  Like  CRN, 
synchronization  is  essential.  The  following  theory  is  paraphrased  from  Law  and  Kelton’ s 
Simulation  Modeling  and  Analysis  (2000): 

Suppose  we  make  n  pairs  of  replications  resulting  in  observations  of  pairs  (Xy , 
X2j), . . . ,  (Xin,  X2n),  where  Xy  is  essentially  Uj  and  X2 /  is  1  -  Uj.  Both  Xy  and  X2j  are 
legitimate  observations  of  the  simulation  model,  therefore 

E[Xu]  =  E[Xv]  =M  (2-6) 

If  each  pair  is  independent  of  the  other  pair,  then  the  total  number  of  replications  is  2 n, 
and 

(X..+X,.) 

X  =  KJZ _ AA  (2.7) 

"  2 

The  average  of  the  A/’s,  X ,  is  the  unbiased  point  estimator  of 

//  =  e[X1.]  =  e[a.]  =  E[a]  (2.8) 

And 

Ya.-r.ri  VarM  Var[xu]  +  Var[zI,]  +  2Ccv[ziJ.A-i,]  (29) 

L  J  n  4/? 
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At  this  point,  like  CRN,  with  no  synchronization  the  covariance  portion  of  equation  (2.9) 
is  zero.  If  we  synchronize  the  pairs  and  induce  a  negative  correlation  between  Xy  and 
Xy,  then 

Cav[xiJ3X2J]  <0  (2.10) 

This  will  result  in  a  lower  Var[xj  ,  the  overall  goal  of  AV  (Law  and  Kelton  2000). 

2.4.3  Control  Variates 

Like  the  previous  two  methods,  “control  variates  attempts  to  take  advantage  of  a 
correlation  between  certain  random  variables  to  obtain  a  variance  reduction”  (Law  and 
Kelton  2000).  However,  this  correlation  in  CV  is  not  induced,  but  already  exists  during 
the  course  of  the  simulation.  Furthermore,  the  sign  of  the  correlation  does  not  matter  for 
CV  (Law  and  Kelton  2000).  Bednar  (2005)  gave  a  complete  explanation  of  the  theory 
behind  CV  in  his  thesis,  Feasibility  Study  of  Variance  Reduction  in  the  Thunder 
Campaign-Level  Model,  derived  from  lecture  notes  given  by  Bauer  (2005)  at  the  Air 
Force  Institute  of  Technology.  His  explanation  includes  a  full  derivation  of  the 
methodology  for  control  variates.  This  full  derivation  is  quoted  from  Bednar  and  found 
in  Appendix  A. 

As  Bednar  (2005)  points  out,  the  selection  of  the  controls  is  of  importance  to  the 
analyst.  The  selection  of  potential  controls  has  two  steps.  First,  the  analyst  must  identify 
an  input  of  a  random  number  or  random  variate.  Then,  the  expected  value  of  the  random 
variate  must  be  determined,  given  the  random  input  parameters  (Bednar  2005).  Bednar 
describes  the  next  steps  in  the  following  paragraph: 

After  all  the  potential  candidates  for  controls  are  identified,  the  average  of  all  the 

realizations  of  each  potential  candidate  must  be  calculated  and  output  in  addition 
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to  the  MOE  [measure  of  effectiveness]  of  interest  for  each  replication.  Since  the 
correlation  between  the  control  candidates  and  the  MOE  is  unknown,  a  stepwise 
regression  must  be  performed  to  identify  which  control  candidates  are 
significantly  correlated  to  the  selected  MOE  (Bednar  2005). 

Once  the  controls  are  identified,  the  analyst  applies  the  steps  in  Appendix  A  to  arrive  at  a 

reduced  confidence  interval  (Bednar  2005). 

2.5.  Previous  Variance  Reduction  Studies 

Variance  reduction  techniques  were  first  introduced  in  the  early  days  of 

computers  using  Monte  Carlo  simulations  (Law  and  Kelton  2000).  The  literature  on  this 

subject  is  extremely  large,  and  the  analysis  of  research  would  take  far  too  long  to  discuss 

in  the  context  of  this  research.  However,  the  application  of  these  variance  reduction 

techniques  is  really  the  focus  of  this  research.  We  have  identified  two  sources  of  research 

that  relate  directly  to  the  subject  of  our  research  effort.  In  1983,  Lieutenant  Colonel 

Mohamed  Elhefny  performed  a  thesis  exploring  the  application  of  different  variance 

reduction  techniques,  comparing  the  results  of  the  various  techniques.  He  concluded, 

“there  is  no  single  technique  which  is  the  most  suitable  technique  for  every  simulation 

problem,”  and  recommended  further  research  to  identify  techniques  effective  for  specific 

simulation  models  (Elhefny  1983).  This  is  significant  because  it  tells  us  that  the  success 

or  failure  of  these  techniques  can  not  be  foreseen  ahead  of  time.  We  must  apply  each 

method  to  the  LCOM  model  to  analyze  the  effectiveness  of  each  method. 

The  Air  Force  Studies  and  Analysis  Agency  sponsored  a  master’s  thesis  by 

Bednar  (2005)  while  at  the  Air  Force  Institute  of  Technology.  He  applied  numerous 

variance  reduction  techniques  to  the  Air  Force’s  THUNDER  combat  simulation  model. 

The  THUNDER  model  is  also  coded  in  SIMSCRIPT  II. 5  programming  language. 
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Bednar  applied  CRN,  control  variates,  antithetic  variates,  and  various  combinations  of  the 
three  methods  in  order  to  detennine  the  most  effective  variance  reduction  techniques  for 
the  THUNDER  model.  His  results  indicated  that  the  control  variates  method  performed 
the  best  of  the  variance  reduction  methods,  while  CRN  and  antithetic  variates  did  not 
produce  successful  results.  When  combining  the  various  techniques,  he  found  that  the 
combination  of  control  variates  and  antithetic  variates  produced  the  most  favorable 
results  (Bednar  2005).  Given  that  both  models  are  written  using  SIMSCRIPT  II. 5,  we 
hope  to  incorporate  the  techniques  used  by  Bednar  in  THUNDER  to  apply  the  same 
variance  reduction  techniques  to  the  LCOM  model. 

2.6.  Random  Number  Generator 

Since  random  number  generators  lie  at  the  root  of  stochastic  simulation,  it  is 
important  that  we  define  and  investigate  the  types  of  random  number  generators  that  we 
will  be  dealing  with  in  our  research  effort.  According  to  Kelton,  Sadowski,  and  Sturrock, 
the  most  common  form  still  built  into  simulation  models  is  a  method  called  the  linear 
congruential  generator  (LCG).  This  method  was  first  introduced  in  1951  by  Lehmer. 
Most  currently  used  LCG’s  have  been  thought  to  produce  a  moduli,  or  cycle  length,  of 
2  -  1,  or  a  little  over  2  billion  numbers.  Once  considered  an  impressive  cycle  length, 
these  numbers  can  be  consumed  in  a  matter  of  minutes  by  today’s  ordinary  PC.  This 
means  the  generator  can  potentially  “lap”  itself  within  a  few  minutes  of  simulation  time, 
given  an  extremely  high  consumption  rate  for  random  numbers  within  the  model. 
Furthermore,  based  on  the  research  of  L’Ecuyer  and  Simard  (2001),  the  poor  structure  of 
the  random  numbers  in  these  types  of  generators  “. .  .can  dramatically  bias  simulation 
results  for  sample  sizes  much  smaller  than  the  period  length”  (L’Ecuyer  et  al.  2002). 
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The  particular  pseudo-random  number  generator  coded  in  SIMSCRIPT  II. 5  is  the 
Lehmer  Pseudo-random  Number  Generator.  The  recursion  is  shown  below  (Lehmer 
1969): 

Xn+\  =  KX„(moA  m)  (2. 1 1) 

This  equation  is  based  on  some  modulus  m  and  multiplier  K.  The  starting  seed  is 
multiplied  by  a  constant  K  to  produce  a  new  seed  and  a  sample  (Lehmer  1969).  This 
generator  performs  like  any  normal  LCG  and  produces  a  cycle  length  of  2  - 1  when  the 
modulus  m  is  a  prime  modulus  and  the  multiplier^  is  one  of  more  than  534  million  full 
period  multipliers  (Law  and  Kelton  2000). 

This  LCG  found  in  SIMSCRIPT  II. 5  gives  the  modeler  10  separate  random 
number  streams  from  which  to  choose  (Russell  1999).  These  10  streams  will  be  used  to 
link  or  synchronize  various  sources  of  randomness  in  the  model  across  multiple 
replications  or  various  scenarios. 

Although  several  alternatives  to  LCG’s  exist  in  the  random  number  generation 
world,  we  will  apply  the  three  variance  reduction  techniques  using  the  current  generator 
found  in  the  SIMSCRIPT  II.5  programming  language. 

2.7.  Minitab  and  Regression 

Minitab  14  is  a  computer  program  designed  to  perform  statistical  functions. 
According  to  the  software’s  website,  Minitab  is  used  by  thousands  of  companies  in  more 
than  80  countries  for  implementation  of  Six  Sigma  and  other  data-driven  quality 
improvement  programs  (Minitab  2007). 

Minitab  14  features  several  methods  for  perfonning  regression  analysis,  such  as 
linear  regression,  polynomial  regression,  logistic  regression,  partial  least  squares, 
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stepwise  and  best  subsets,  and  residual  plots  (Minitab  2007).  The  stepwise  regression 
analysis  is  used  to  investigate  and  model  the  relationship  between  a  response  variable  and 
two  or  more  predictors.  Stepwise  regression  removes  and  adds  variables  to  the  regression 
model  for  the  purpose  of  identifying  a  useful  subset  of  the  predictors,  using  what  is  called 
a  forward  selection  to  add  variables  and  then  a  backward  elimination  to  remove  variables 
(Minitab  2007).  The  user  simply  specifies  the  response  variable,  the  starting  set  of 
predictor  variables,  and  the  alpha  value  for  adding  or  removing  a  variable  to  or  from  the 
model.  The  following  is  paraphrased  from  the  explanation  of  the  stepwise  procedure  as 
stated  in  the  Minitab  user  manual: 

The  first  step  in  stepwise  regression  is  to  calculate  an  /-’-statistic  and  /7-value  for 
each  variable  in  the  model.  If  the  model  contains  n  replications  where  j  =  1,2,. .  .n  ,  then  F 
for  any  variable,  xj,  is 


F, 


(SSEU_X„+SSE,) 


MSEj 


(2.12) 


where  SSEq .  xj,  =  Sum  of  Squared  Error  for  the  model  that  does  not  contain  xh  and  SSEj  = 
Sum  of  Squared  Error  and  MSEj  =  Mean  Square  Error  for  the  model  that  contains  xr  For 
the  backwards  elimination,  if  the  /7-value  for  any  variable  is  greater  than  the  value 
specified  in  a  to  remove,  then  Minitab  removes  the  variable  with  the  largest  /7-value.  If 
Minitab  cannot  remove  a  variable,  the  procedure  attempts  to  add  a  variable.  Minitab 
calculates  an  /•’-statistic  and  /7-value  for  each  variable  that  is  not  in  the  model.  If  the 
model  contains  j  variables,  then  F  for  any  variable,  Xj,  is 
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(2.13) 


F, 


(i.H-y-i) 


(SSE^-SSE,^,) 
MSE(j  +  xj) 


where  n  =  number  of  observations,  SSE=  Sum  of  Squared  Error  before  xj  is  added  to  the 
model,  and  SSE(j  +Xj)  =  Sum  of  Squared  Error  and  MSEq  +Xj)  =  Mean  Square  Error  after  Xj 
is  added  to  the  model.  For  the  forward  selection,  if  the  /?-value  corresponding  to  the  F- 
statistic  for  any  variable  is  smaller  than  the  value  specified  in  a  to  enter,  Minitab  adds  the 
variable  with  the  smallest  /7-value  to  the  model.  When  no  more  variables  can  be  entered 
into  or  removed  from  the  model,  the  stepwise  procedure  ends  (Minitab  2007). 

2.8.  Analysis 

Several  statistical  methods  will  be  used  to  analyze  the  LCOM  output  and  determine  the 
effectiveness  of  each  technique  when  applied  to  the  model.  The  following  sections 
describe  these  methods. 

2.8.1.  Confidence  Intervals 

For  single  model  tests  such  as  control  variates,  we  will  construct  confidence 
intervals  for  the  mean  estimate  using  the  following  equation: 

CI=  X  +/-tal2,n-i(S/^)  (2-14) 

In  this  equation,  ju  is  the  mean,  t  is  the  quartile  of  the  t  distribution  with  n- 1  degrees  of 
freedom,  S  is  the  standard  deviation,  and  n  is  the  number  of  replications.  This  equation 
assumes  the  output  for  the  mean  follows  a  normal  distribution  (Banks  et  al.  2005).  A 
reduction  in  Cl  width  will  be  measured  for  a  successful  reduction  of  variance. 
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2.8.2.  Determining  the  Number  of  Replications 


In  order  to  detennine  the  number  of  replications  to  achieve  a  desired  halfwidth  [1 
we  will  apply  the  following  equation  (McClave  et  al.  2005): 

tKnfv1 


n  = 


0 se y 


(2.15) 


In  equation  2. 15,  SE  is  known  as  the  sampling  error.  In  this  case,  the  sampling  error  is 
equal  to  the  half-width,  /?,  of  the  confidence  interval  desired  (McClave  et  al.  2005).  A 
significant  reduction  in  the  number  of  replications  required  will  indicate  a  successful 
application  of  a  particular  variance  reduction  technique  (Bednar  2005). 

2.8.3.  Paired-t  Confidence  Interval 

Finally,  a  paired  t-test  will  be  used  to  calculate  a  confidence  interval  about  the 
difference  between  two  values.  This  will  tell  us  if  the  changes  in  the  model  result  in  a 
statistically  significant  values.  To  form  the  100(1-  a)  confidence  interval  we  use 
X\j  -  X2j  =  Z,  for  j  =  1  to  n.  Therefore,  the  following  equation  for  the  confidence  interval 
applies  (Law  and  Kelton  2000): 

CI[Z]  =  Z±W1^_Z_  (2.16) 


If  the  confidence  interval  includes  zero  we  can  not  conclude  that  the  two  values  are 
statistically  significant  at  the  specified  a  level,  and  the  variance  reduction  technique  has 
been  ineffective  in  reducing  the  variance  of  the  output. 
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3.  Methodology 


This  chapter  begins  with  the  general  research  process  for  each  variance  reduction 
method.  It  then  outlines  the  selection  of  the  particular  model  used  in  the  analysis,  and  the 
selection  of  output  measures.  This  is  followed  by  a  description  of  the  methodology 
behind  the  application  of  each  variance  reduction  technique  on  the  model.  The  chapter 
concludes  with  a  discussion  of  the  collection  of  output  data  through  model  runs  and  the 
preparation  of  the  data  for  analysis. 

3.1.  Research  Process 

The  objective  of  this  research  is  to  study  the  impact  of  various  variance  reduction 
techniques  on  LCOM.  The  research  will  apply  these  techniques  to  the  model  and  analyze 
the  results,  determine  whether  or  not  each  technique  is  effective  and  identify  the  most 
effective  method.  The  accomplishment  of  this  objective  is  achieved  through  the 
following  general  steps: 

1 .  Identify  a  particular  measure  of  effectiveness  (MOE)  or  set  of  MOE’s  in  the 
model’s  output  that  are  commonly  used  by  LCOM  users 

2.  Identify  the  locations  in  the  model  where  random  numbers  are  generated 

3.  Alter  the  model’s  source  code  in  order  to  apply  the  desired  variance  reduction 
technique 

4.  Analyze  the  results  from  each  variance  reduction  technique  to  detennine  the 
effectiveness  of  each  technique 

For  the  purpose  of  this  research,  the  locations  in  the  model  where  random  numbers  are 
generated  and  consumed  will  be  referred  to  as  random  variate  draws  or  points  of 
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consumption.  The  two  terms  will  be  used  synonymously  throughout  the  next  three 
chapters. 

3.2.  Model  Selection  and  Parameters  for  LCOM  Analysis 

Development  of  an  actual,  real-world  scenario  for  LCOM  can  take  many  months 
to  develop  (Erdman  2006).  Fortunately,  the  LCOM  program  includes  two  example 
scenarios  for  training  and  education.  The  first  scenario,  the  Bicycle  Model,  is  a  very 
simplistic  model  involving  a  bicycle  used  for  delivering  papers  early  every  morning.  The 
second  scenario,  the  Joint  Service  FX-99  Generic  Fighter  Model,  is  loosely  based  on  an 
F-16  aircraft  maintenance  unit,  with  information  compiled  from  data  at  Hill  AFB  from 
July  1979  to  June  1980  (ASC/ENM  2004).  Since  the  reliability  and  maintainability 
parameters  can  be  quickly  changed  and  updated  for  generic  applications,  the  name  of  the 
model  has  evolved  to  be  known  as  the  Generic  Fighter  Model  (ASC/ENM  2004). 

Unfortunately,  the  simplicity  of  the  bicycle  model  is  such  that  there  is  little  or  no 
variance  in  the  original  model,  so  the  application  of  variance  reduction  techniques  would 
obviously  show  no  significant  improvement  over  the  original.  On  the  other  hand,  the 
generic  fighter  model  is  much  more  realistic  and  similar  to  current  LCOM  models  in  use 
today.  Additionally,  the  variance  in  the  output  is  sufficiently  large  enough  for  a  reduction 
in  the  variance  to  be  possible  and  desirable.  For  these  reasons,  the  Generic  Fighter 
Model  was  selected  for  analysis  in  this  study. 

The  original  configuration  of  the  generic  fighter  model  is  sufficient  for  both  AV 
and  CV,  so  the  default  configuration  was  used  for  these  techniques.  However,  since 
CRN  requires  the  comparison  between  two  different  configurations,  a  modification  of  the 
model  for  this  technique  was  necessary.  To  modify  the  model  in  a  way  that  would  have  a 
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significant  effect  on  the  outcome  of  the  MOE’s,  the  manpower  availability  for  the  FLTL 
manpower  type  was  adjusted  and  the  model  run  with  a  30-replication  production  run  until 
a  significant  difference  in  the  two  output  statistics  of  interest  were  empirically  observed. 
The  output  statistics  of  interest  will  be  explained  in  detail  in  the  next  section.  Then,  a 
paired  difference  test  of  hypothesis  was  conducted  for  jUd=  (pi  -  p 2)  in  order  to  verify  that 
the  mean  for  the  two  scenarios  are  not  equal  (for  results  and  calculations  of  paired 
difference  test  see  Appendix  B).  In  the  original  model,  parts  and  labor  are  both 
essentially  unlimited.  For  the  second  scenario,  the  model  simply  contains  a  constrained 
value  for  FFTF  manpower  of  50  workers  available  for  each  shift. 

The  period  of  interest  for  this  experiment  consisted  of  a  5 -day  period  following  a 
20-day  warm-up  period.  After  consulting  with  FCOM  users  it  was  detennined  that  a  20- 
day  wann-up  was  likely  to  be  sufficient  for  the  model  to  exhibit  a  steady-state  behavior, 
based  upon  their  experiences  and  recommendations  (Erdman  2006).  Additionally,  a  5- 
day  observation  period  from  day  2 1  to  day  25  would  provide  a  significant  amount  of 
variance  before  the  implementation  of  the  variance  reduction  techniques. 

3.3.  Output  Measure  Selection 

The  FCOM  generates  dozens  of  statistics  in  its  output,  with  most  calculations 
typically  experiencing  some  sort  of  variance  across  multiple  replications.  However,  after 
speaking  with  expert  FCOM  users,  the  MOE  that  interests  the  users  the  most  is  the 
statistic  labeled  C15  on  the  output  tool,  known  as  Overall  Achieved  Sorties  per  Aircraft 
per  Day  (Erdman  2006).  This  output  data  is  located  in  the  production  run  merged  output 
report.  An  example  of  a  typical  merged  output  report  for  the  Bicycle  Model  is  contained 
in  Appendix  C. 
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LCOM  automatically  collects  and  calculates  the  infonnation  for  these  statistics. 
Although  LCOM  generates  several  dozens  of  various  statistics,  the  decision  was  made  to 
analyze  the  variance  of  two  output  statistics  in  particular,  C15  -  Overall  Achieved  Sorties 
per  Aircraft  per  Day,  and  C24  -  Mission  Capable  Rate.  This  decision  was  reached  for 
several  reasons.  Since  LCOM  users  focus  almost  exclusively  on  the  C15  statistic,  the 
reduction  in  the  variance  of  any  other  statistic  separate  from  C15  would  have  little  or  no 
impact  on  the  way  the  model  was  run  and  analyzed.  Additionally,  Mission  Capable  Rate 
has  a  direct  impact  on  the  sortie  generation  rate,  so  C24  was  also  included.  Second,  the 
Generic  Fighter  Model  produces  a  fair  amount  of  variance  in  both  output  statistics  C15 
and  C24,  making  variance  reduction  a  feasible  and  desirable  goal  for  our  research  and 
analysis. 

3.4.  Common  Random  Numbers 

As  noted  previously,  the  random  number  generator  in  SIMSCRIPT  II. 5  allows  the 
user  to  identify  up  to  10  unique  random  number  streams.  In  the  LCOM’s  source  code  for 
the  main  model,  33  separate  random  variate  draws  or  points  of  consumption  were 
identified.  In  the  original  source  code,  LCOM  uses  only  the  first  9  of  the  10  available 
random  number  streams.  The  model’s  random  number  generators  are  organized  by 
function,  dedicating  a  stream  to  different  tasks  in  the  manner  shown  in  Table  3.1 
(ASC/ENM  2004): 
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Table  3.1:  LCOM  Random  Number  Streams  and  Associated  Functions 


Stream 

Function 

1 

Attribute  initial  values.  %  sortie  to  complete  (AABORT  &  ATTRIT)  Ram  time 

2 

Task  Durations 

3 

Failure  Clock  Operations 

4 

Time  Accumulating  Attributes  Random  Setting 

5 

Probability  of  Air  Abort.  Attrition,  or  Ram  Repair 

6 

Task  Selection  on  A.  E.  and  G  Selection  Modes 

7 

Random  Multiplier  for  Initial  Failure  Clock  Settings 

8 

Sortie  Length  (Task  Time  Option) 

9 

Not  used 

10 

Unknown 

LCOM  allows  the  user  to  specify  or  create  starting  seeds  for  each  of  the  first  8  streams 
specified  above,  using  the  LCOM  interface  screen  shown  in  Appendix  D.  However, 
several  different  random  variate  draws  within  the  model  sampled  from  the  same  stream. 
This  means  that  as  the  model’s  inputs  change,  the  one-for-one  relationship  between 
random  variates  across  different  scenarios  is  lost.  Even  with  the  ability  to  set  seeds  for 
each  stream  and  synchronize  the  random  number  streams,  the  lack  of  a  one-for-one 
relationship  between  random  variates  consumed  across  different  scenarios  makes  the 
application  of  CRN  unlikely  to  be  successful  or  consistent  in  the  pursuit  of  variance 
reduction. 

Each  of  the  33  points  of  consumption  in  the  model  was  investigated  one-by-one  to 
determine  the  relative  use  of  each  point  in  the  model  where  random  numbers  were 
generated  and  consumed.  After  investigating  the  33  random  variate  draws  in  the  LCOM 
main  model  source  code,  it  was  detennined  that  8  random  variate  draws  were  in  use  for 
FX99  model.  In  other  words,  only  8  points  of  consumption  in  the  default  settings  of 
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FX99  actively  generated  and  consumed  random  numbers.  The  remaining  25  points  of 
consumption  were  completely  inactive.  The  random  variate  draws  in  use  usually  shared 
the  same  random  number  stream.  Figure  3.1  illustrates  the  total  number  of  random 
variate  draws  assigned  to  each  random  number  stream  in  the  main  model  code. 
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Figure  3.1:  Configuration  of  Random  Variate  Draws  by  Stream 

The  code  was  modified  so  that  each  active  random  variate  draw  was  synchronized  by 
identifying  a  unique  random  number  stream  to  each  of  the  points  in  the  model.  The 
modified  configuration  of  random  variate  draws  is  shown  in  Figure  3.2,  with  the 
modified  random  variate  draws  shown  in  red. 
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Figure  3.2:  Modified  Configuration  of  Random  Variate  Draws  by  Stream 

Since  less  than  8  of  the  33  random  variate  draws  are  active  in  the  the  FX99  model,  the 
current  configuration  of  LCOM  allowed  the  synchronization  of  the  random  variate  draws 
in  use  by  simply  identifying  the  same  seed  set  across  scenarios  in  the  ISEEDS  tab  of  the 
production  run  interface  shown  in  Appendix  D.  Then,  as  an  additional  precaution,  the 
remaining  25  inactive  points  of  consumption  were  split  to  share  the  remaining  9th  and  10th 
random  number  streams. 

Once  the  source  code  was  modified  to  reflect  these  changes,  the  code  was 
compiled  using  Simscript’s  SimStudio  compiling  environment  and  an  LCOM  production 
run  of  30  replications  was  conducted  with  the  new,  modified  source  code  using  the 
default  scenario  with  unconstrained  manpower.  Then,  the  scenario  was  changed  to  the 
second  scenario  with  constrained  FLTL  manpower  resources  and  the  30-replication 
production  run  was  completed  again.  The  paired  difference  from  each  replication  was 
used  to  create  the  confidence  interval  and  halfwidth. 
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Similarly,  the  unmodified  model  was  run  and  a  halfwidth  calculated  for  both  the 
unconstrained  manpower  scenario  and  the  constrained  manpower  scenario  in  order  to 
provide  a  base  from  which  to  compare  the  new  results.  This  paired  difference  across  the 
two  different  scenarios  was  used  to  calculate  the  confidence  interval  and  halfwidth  in  the 
exact  same  manner  as  the  CRN  model  described  above.  The  reduction  in  the  confidence 
interval  halfwidth  was  calculated  to  determine  the  degree  of  variance  reduction. 

3.5.  Antithetic  Variates 

Whereas  CRN  requires  a  comparison  between  two  different  scenarios,  antithetic 
variates  relies  on  an  induced  correlation  among  replications  within  the  same  scenario. 
Since  no  additional  scenarios  are  required  for  this  technique,  the  default  settings  for  the 
48-aircraft  Generic  Fighter  Model  were  used. 

Recall  that  if  Uk  is  a  particular  random  number  used  for  a  particular  purpose  in  the 
first  run,  then  1  -  Uk  is  used  for  the  same  purpose  in  the  second  run.  Fortunately, 
SIMSCRIPT  II. 5  allows  the  incorporation  of  antithetic  variates  with  some  very  simple 
modifications.  According  to  the  manual,  Building  Simulation  Models  with  SIMSCRIPT 

11. 5,  “To  use  an  antithetic  variate  in  any  random  deviate  generator  in  SIMSCRIPT  II. 5,  it 
is  merely  necessary  to  negate  the  random  number  stream  parameter  to  the  function 
random,  f ”  (Russell  1999).  Random.f  is  the  SIMSCRIPT  II. 5  function  for  a  random 
number  drawn  between  0  and  1 .  For  example,  a  portion  of  the  LCOM  source  code  is 
shown  in  Figure  3.1: 

LET  VALUE. ATB ( OWNER, SEQ, TAIL, 3)  = 

_ REAL. F(INT.F (RANDOM.F (4) ) ) 

Figure  3.3:  LCOM  Random  Variate  Example 
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The  random  variate  in  this  example,  RANDOM .  F  ( 4 ) ,  is  a  pseudo-random  number 
uniformly  distributed  between  0  and  1,  drawn  from  SIMSCRIPT  II. 5’ s  random  number 
stream  4.  In  order  to  implement  antithetic  variates,  the  portion  of  the  code 
RANDOM .  F  ( 4 )  simply  becomes  RANDOM .  F  ( -  4 )  in  order  to  turn  Uk  into  1  -  Uk 
(Russell  1999)  .  Every  point  in  the  model  where  the  function  random.f  is  used,  the 
random  number  stream  parameter  must  be  negated.  This  includes  all  33  random  variate 
sources,  or  points  of  consumption  in  the  model. 

The  problem  of  random  variates  must  also  be  considered.  Some  sources  of 
randomness  in  LCOM  draw  from  commonly  used  distributions,  such  as  normal, 
lognormal,  exponential,  and  poisson.  These  points  in  the  model  typically  appear  with  the 
distribution  name,  distribution  parameters,  and  stream  number  from  which  to  sample. 

For  example,  a  random  number  drawn  from  a  normal  distribution  may  appear  as 
normal .  f  (MU,  SIGMA,  3 ) ,  with  stream  3  specified  for  this  draw.  In  this  case,  Uk 
is  still  a  random  number  between  0  and  1,  generated  using  random.f  before  SIMSCRIPT 
II. 5  converts  the  number  according  to  the  distribution  and  parameters.  Consequently,  like 
the  random  variates  created  using  random.f,  the  stream  is  simply  negated  and 
SIMSCRIPT  II. 5  conducts  the  process  of  turning  1  -  Uk  into  the  appropriate  random 
deviate,  given  the  distribution  and  specified  parameters  (Russell  1999). 

Like  CRN,  the  new  code  is  modified  to  reflect  the  change  at  each  of  the  33  points 
of  consumption  in  the  model,  compiled  in  SimStudio,  and  run  using  both  the  unmodified 
code  as  well  as  the  modified  code  with  antithetic  variates.  Like  CRN,  a  one-for-one 
relationship  between  Uk  and  1  -  Uk  is  essential  to  synchronize  the  model  and  induce  the 
correlation  required  to  achieve  a  reduction  in  variance.  For  this  reason,  careful  attention 
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must  be  paid  to  making  sure  the  starting  seeds  for  each  stream  remained  the  same  for 
both  the  original  model  and  the  model  with  the  antithetic  variates.  In  this  case,  unlike 
CRN,  the  success  of  the  model  depends  on  the  ability  to  ensure  a  one-for-one  relationship 
across  models  without  changing  the  scenario  parameters.  Consequently,  the  starting 
seeds  are  simply  created  and  identified  for  each  replication  using  the  ISEEDS  tab  of  the 
production  run  portion  of  the  LCOM  graphical  user  interface  (see  Appendix  D).  A 
production  run  of  15  replications  was  made  with  the  unmodified  code  and  15  with  the 
antithetic  variates  code.  The  output  for  both  productions  runs  were  combined  and 
averaged  in  order  to  obtain  the  new  confidence  interval  calculation  and  halfwidth.  The 
results  were  then  compared  to  a  30-replication  production  run  with  the  unmodified 
model. 

3.6.  Control  Variates 

The  Control  Variate  method  relies  on  the  relationship  between  a  random  variate 
input  and  the  output  variable  of  interest.  This  correlation  along  with  the  deviation  from 
the  known  expected  value  of  the  input  variable  are  used  to  adjust  the  output  variable  up 
or  down,  closer  to  the  true  but  unknown  mean.  In  LCOM,  the  number  of  potential 
controls  can  reach  the  hundreds.  However,  each  potential  control  must  be  considered  by 
capturing  each  random  variate  value  drawn  in  the  model,  along  with  the  expected  value, 
distribution  type  and  parameters  over  the  course  of  the  5-day  period. 

The  first  step  in  this  process  involved  modifying  the  LCOM  main  model  source 
code  in  order  to  capture  each  random  variate  drawn  and  consumed  in  the  model. 
Capturing  the  random  variates  required  the  use  of  the  “print”  command  to  dump  the 
random  variates  in  an  output  file  called  a  PSR  report  for  each  replication  in  the  LCOM 
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production  run.  Two  different  examples  of  modified  random  variate  draws  are  shown  in 
Figure  3.2,  the  first  from  a  logical  expression  and  the  second  through  a  subroutine: 


IF  VALUE  EQ  0 

LET  VALUE  =  RANDI . F ( 1, 100000,  6) 
print  1  line  with  value  thus 
****** 


LET  DURATION  =  DRAW (D 1ST, FIRST, SECOND, 2 , TYPE . TASK (TDIM1, TDIH2 ) ) 
print  1  line  with  duration,  dist,  first,  second  thus 
***,*****  **  ******,*******  *****_****** 
always 

Figure  3.4:  Two  Modified  Random  Variate  Draws  for  Control  Variates 


In  the  first  case,  RANDI.F  is  a  command  that  generates  a  discrete  random  integer 
between  1  and  100000  using  stream  number  6.  In  the  second  case,  DRAW  references  a 
routine  that  generates  a  random  number  based  on  the  variables  listed  in  parentheses.  For 
the  purposes  of  this  exercise,  the  first  four  variables  are  the  only  variables  of  significance. 
The  first  variable  is  the  distribution  type,  the  second  is  the  first  parameter  of  the  specific 
distribution,  the  third  is  the  second  parameter,  and  the  fourth  is  the  random  number 
stream.  All  random  variates  used  in  LCOM  were  modified  in  the  same  manner  as  the 
two  shown  in  Figure  3.2. 

Once  in  the  PSR  report,  the  random  variates  were  cut  and  pasted  to  an  Excel 
spreadsheet  where  the  numbers  were  sorted  and  separated  by  distribution  type  and 
parameter.  Once  sorted,  each  unique  distribution  formed  the  basis  for  a  potential  control 
to  be  analyzed.  This  means  that  different  distribution  types  and  parameters  could  come 
from  a  single  random  variate  draw.  The  total  number  of  potential  controls  was  23.  After 
eliminating  those  potential  controls  with  no  random  variates  drawn  in  any  single 
replication,  the  number  of  potential  controls  decreased  to  20.  In  actuality,  13  of  the  20 


32 


potential  controls  came  from  a  single  random  variate  draw  in  the  task  duration  function. 
Since  regression  theory  requires  at  least  as  many  replications  as  potential  controls,  2 1 
replications  were  performed  (McClave  et  al.  2005). 

Calculations  for  each  of  the  2 1  replications  were  performed  in  Excel,  recording 
the  control  number,  the  first  parameter  (Parameterl),  the  second  parameter  (Parameter2), 
the  observed  mean  (Value),  and  the  calculated  difference  between  the  expected  and 
observed  means.  Finally,  the  output  statistics  C15  and  C24  for  each  replication  were 
recorded.  An  example  is  shown  in  Table  3.4: 


Table  3.2:  Potential  Control  Observations,  Single  Replication 


Control 

Parameterl 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004296 

-0.00013 

2 

0.007083 

0.002083 

0.006909 

0.000174 

3 

0.008333 

0.0025 

0.008226 

0.000107 

4 

0.010417 

0.002917 

0.010755 

-0.00034 

5 

0.01 166667  0.002916667  0.012967 

-0.0013 

6 

0.0125 

0.00375 

0.011788 

0.000712 

7 

0.012917 

0.00375 

0.013393 

-0.00048 

8 

0.020833 

0.005833 

0.020451 

0.000382 

9 

0.02125 

0.005833 

0.022362 

-0.00111 

10 

0.025 

0.007083 

0.025813 

-0.00081 

11 

0.033333 

0.009583 

0.033729 

-0.0004 

12 

0.034167 

0.009583 

0.03227 

0.001897 

13 

0.041667 

0.024167 

0.037604 

0.004063 

14 

50000.5 

3000 

52359.79 

-2359.29 

15 

50000.5 

6000 

48885.22 

1115.282 

16 

50000.5 

10000 

47602.14 

2398.362 

17 

50000.5 

15000 

48855.5 

1144.995 

18 

50000.5 

50000 

47690.16 

2310.335 

19 

50000.5 

75000 

60801.71 

-10801.2 

20 

50000.5 

1-100000 

49346.3 

654.2028 

C15 

1.72 

C24 

67.91 
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A  copy  of  the  observations  for  each  replication  can  be  seen  in  Appendix  F.  Controls  1-13 
are  all  lognormally  distributed,  with  the  expected  mean  in  Parameter  1  and  the  expected 
standard  deviation  in  Parameter  2.  Controls  14-20  are  all  unifonn,  discrete  random 
variables  between  1  and  100,000.  In  the  case  of  controls  14-20,  Parameterl  refers  to  the 
expected  mean  and  Parameter2  refers  to  a  cutoff  point  for  decision  tree  task  selection. 

The  differences  between  the  observed  and  expected  means  for  each  potential 
control  were  arranged  in  a  20  X  2 1  matrix  and  exported  to  Minitab  for  the  stepwise 
regression  calculation  (See  Appendix  G).  The  predictors  identified  in  the  stepwise 
regression  automatically  became  the  control  variates  used  to  perfonn  the  calculations 
listed  in  Appendix  A,  ultimately  concluding  with  a  new,  smaller  confidence  interval  and 
halfwidth. 

3.7.  Data  Collection 

When  conducting  a  production  run  in  LCOM,  the  user  specifies  the  number  of 
replications  desired  for  the  run  using  the  screen  shown  in  Appendix  E.  After  the  run,  the 
output  from  each  replication  is  stored  in  a  PSR  report.  One  PSR  report  is  created  for  each 
replication.  Then,  a  merged  post  processor  merges  statistics  from  the  individual  PSR 
statistics  files  output  over  all  replications  of  the  main  model  specified  in  the  production 
run  (ASC/ENM  2004).  The  output  of  this  file,  shown  in  Appendix  B,  includes  the  mean 
value,  the  standard  deviation,  minimum  and  maximum  values  across  replications,  and 
automatically  computes  a  95%  confidence  interval  for  each  statistic  (ASC/ENM  2004). 
For  the  purpose  of  this  research,  the  confidence  interval,  mean,  and  standard  deviation  are 
used  primarily  for  the  analysis  of  each  technique.  Additionally,  the  output  statistic  for 
each  replication  must  be  obtained  for  use  in  the  paired  difference  when  analyzing  the 
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common  random  numbers  technique.  This  information  is  obtained  from  the  PSR  file  for 
each  individual  replication.  Likewise,  all  random  variate  draws  were  coded  in  the  main 
model  source  code  to  dump  to  the  PSR  file  after  each  individual  replication. 

3.8.  Analysis 

Chapter  2  discusses  the  statistical  calculations  involved  in  the  analysis  of  this 
experiment.  These  calculations  include  confidence  intervals  about  the  mean,  confidence 
intervals  about  a  difference  between  two  values,  and  replications  for  a  desired  halfwidth. 
The  calculations  are  used  to  detennine  the  variance  reduction  achieved  by  each 
technique. 

First,  for  the  base,  unmodified  model,  a  confidence  interval  about  the  mean  was 
identified  and  a  confidence  interval  halfwidth  was  calculated  using  the  results  of  a 
production  run  of  30  replications.  For  techniques  involving  a  single  scenario,  such  as 
antithetic  variates  and  control  variates,  a  confidence  interval  about  the  mean  was 
identified  for  each  of  the  two  techniques  in  the  merged  output  report,  along  with  the 
calculation  of  the  new  confidence  interval  halfwidths.  The  percent  improvement  over  the 
base  model  in  the  confidence  interval  halfwidth  achieved  by  the  each  variance  reduction 
technique  detennines  the  degree  of  variance  reduction. 

Additionally,  if  any  level  of  variance  reduction  is  observed  in  the  output,  the 
improvement  can  be  approximated  by  determining  the  number  of  additional  replications 
required  by  the  unmodified  model  in  order  to  achieve  the  halfwidth  observed  in  the 
model  with  the  variance  reduction  technique  applied.  This  demonstrates  the  additional 
effort  required  to  achieve  the  precision  of  the  new  model. 


35 


For  the  common  random  numbers  technique,  the  base  model  must  include  the 
same  two  scenarios  included  in  the  CRN  models.  Recall  that  the  modification  included 
unconstrained  and  constrained  manpower  scenarios.  The  CRN  code  and  the  unmodified 
code  were  compiled  and  run  for  the  two  different  scenarios.  Then,  a  paired  t-test  was 
used  to  calculate  the  confidence  interval  of  the  difference  between  the  two  values  across 
the  different  scenarios.  This  calculation  included  the  differences  in  the  two  scenarios  for 
both  the  new,  CRN  code  and  the  unmodified  source  code.  Now,  using  the  same  method 
as  the  other  two  techniques,  the  confidence  interval  halfwidths  were  calculated  and  the 
percent  improvement  determined  the  degree  of  variance  reduction. 
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4.  Results  and  Analysis 


This  section  contains  results  and  analysis  performed  to  determine  the 
effectiveness  of  each  variance  reduction  technique  applied  to  LCOM.  The  chapter 
describes  the  results  achieved  from  common  random  numbers,  antithetic  variates,  and 
control  variates. 

4.1.  Common  Random  Numbers 

The  common  random  numbers  model  was  compared  to  the  base  model  using  the 
difference  between  the  two  scenarios  with  constrained  and  unconstrained  manpower  as 
specified  in  Chapter  3.  The  full  results  for  each  replication  can  be  seen  in  Appendix  H. 
A  summary  of  the  results  for  the  C15  statistic,  Overall  Achieved  Sorties  per  Aircraft  per 
Day,  with  the  confidence  interval  (Cl)  calculated  about  the  mean  difference  from 
replication  to  replication  between  the  two  scenarios,  is  shown  in  Figure  4.1: 

Base  —  C15 

95%  Cl:  0.215352775  0.26398056 

Cl  halfwidth:  0.024313892 
CRN  —  C15 

95%  Cl:  0.1669436580.23305634 

Cl  halfwidth:  0.033056342 
Improvement:  -35.96% _ 

Figure  4.1:  Common  Random  Numbers  Results  Summary,  C15  Statistic 

Likewise,  the  same  calculations  were  conducted  for  the  models  using  the  C24  statistic, 
Mission  Capable  Rate,  as  the  variable  of  interest.  A  summary  of  the  results  for  the  C24 
statistic  is  shown  in  Figure  4.2: 


37 


Base  —  C24 

95%  Cl:  3.3470195694.78031376 

Cl  halfwidth:  0.716647098 
CRN  -  C24 

95%  Cl:  3.8258380995.15482857 

Cl  halfwidth:  0.664495234 
Improvement: _ 7.28% _ 

Figure  4.2:  Common  Random  Numbers  Results  Summary,  C24  Statistic 

A  negative  improvement  indicates  an  increase  in  the  confidence  interval  size,  the 
opposite  of  the  desired  effect  in  this  experiment.  As  Figure  4. 1  shows,  the  confidence 
interval  for  the  C15  statistic  did  not  improve  from  the  base  model  to  the  CRN  model.  In 
fact,  the  confidence  interval  halfwidth  was  significantly  larger  in  the  CRN  model.  On  the 
other  hand,  the  confidence  interval  halfwidth  for  the  C24  statistic  improved  slightly  by 
7.28  percent. 

To  put  this  improvement  in  the  C24  variance  into  perspective,  equation  2.15  was 
used  to  determine  the  number  of  replications  required  by  the  original  model  in  order  to 
achieve  the  same  confidence  interval  halfwidth  observed  in  the  CRN  model.  In  order  to 
achieve  the  same  confidence  interval  halfwidth  as  the  CRN  model,  the  user  would  need 
to  run  approximately  35  replications  in  order  to  achieve  the  same  confidence  in  the  C24 
output  statistic,  versus  30  replications  with  the  CRN  model. 

4.2.  Antithetic  Variates 

The  results  for  antithetic  variates  were  even  less  favorable  than  the  common 
random  numbers  results.  In  this  experiment,  the  base,  unmodified  FX99  model  was  run  a 
total  of  30  replications.  Then,  the  main  model  code  was  modified  to  incorporate  the 
antithetic  variates  at  each  point  in  the  model  where  a  random  variate  draw  occurs.  The 
code  was  compiled  and  the  FX99  model  was  run  again  for  a  total  of  30  replications,  15 
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with  Uj  and  15  with  1  -Uj.  The  full  results  can  be  seen  in  Appendix  I.  A  summary  of  the 
results  for  the  C15  output  statistic  is  shown  in  Figure  4.3: 


Base  -Cl  5 

95%  Cl:  2.174569922.230763413 

Cl  halfwidth:  0.02809675 
Antithetic  Variates  —  C15 
95%  Cl:  2.145646262.216353742 

Cl  halfwidth:  0.03305634 
Improvement:  -25.83% _ 

Figure  4.3:  Antithetic  Variates  Results  Summary,  C15  Statistic 

Like  the  common  random  numbers  experiment,  the  confidence  interval  for  the  C15 
output  statistic  does  not  improve,  but  actually  worsens  by  more  than  25  percent.  Similar 
results  were  observed  in  the  C24  output  statistic  in  this  experiment.  A  summary  of  the 
results  is  shown  in  Figure  4.4: 


Base  —  C24 

95%  Cl:  66.593192167.21947454 

Cl  halfwidth:  0.31314121 
Antithetic  Variates  —  C24 
95%  Cl:  66.766296267.59970375 

Cl  halfwidth:  0.41670375 
Improvement:  -33.07% _ 

Figure  4.4:  Antithetic  Variates  Results  Summary,  C24  Statistic 

Contrary  to  CRN,  the  halfwidth  for  the  C24  output  statistic  got  much  worse  after  the 
implementation  of  the  antithetic  variates. 

It  is  apparent  that  the  synchronization  techniques,  common  random  numbers  and 
antithetic  variates,  do  not  improve  the  variance  of  the  output  statistics.  In  fact,  the 
improvement  observed  in  the  variance  of  C24  in  the  CRN  model  more  than  likely  occurs 
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due  to  the  randomness  of  the  model,  and  not  due  to  any  true  synchronization.  When 
examining  the  random  variate  draws  in  the  model  and  the  purpose  of  each  specific  draw, 
it  is  easy  to  understand  why  synchronization  is  not  possible  the  way  the  model  is 
currently  constructed.  For  example,  task  durations  all  originally  sampled  from  random 
number  stream  2.  It  was  determined  that  the  task  durations  were  generated  using  three 
different  random  variate  draws  in  different  places  in  the  model.  However,  upon  further 
examination,  the  vast  majority  of  task  durations  are  all  generated  and  consumed  from  a 
single  random  variate  draw.  For  a  maintenance  unit,  this  includes  all  scheduled  and 
unscheduled  task  times  generated  for  every  flightline,  backshop,  and  depot  task, 
regardless  of  the  priority.  In  the  current  configuration  of  LCOM,  essentially  all  tasks 
sample  from  a  single  point  in  the  model.  As  discussed  in  Chapter  3,  even  with  the  ability 
to  set  seeds  for  each  stream  and  synchronize  the  random  number  streams,  the  lack  of  a 
one-for-one  relationship  between  random  variates  consumed  across  different  scenarios 
renders  the  current  configuration  useless  for  the  application  of  both  common  random 
numbers  and  antithetic  variates.  If  the  individual  tasks  are  not  separated,  the  one-for-one 
relationship  between  random  variates  consumed  across  different  scenarios  is  not 
achievable. 

4.3.  Control  Variates 

Unlike  the  previous  two  methods,  the  control  variates  method  does  not  attempt  to 
induce  a  correlation  in  the  model.  Instead,  the  control  variates  method  attempts  to  take 
advantage  of  a  known  correlation  between  a  random  variate  and  the  output  variable  of 
interest. 
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As  described  in  Chapter  3,  and  similar  to  the  other  two  experiments,  this  method 
was  applied  to  two  different  output  variables  of  interest,  C15  -  Overall  Achieved  Sorties 
per  Aircraft  per  Day,  and  C24  -  Mission  Capable  Rate.  Since  20  different  potential 
controls  were  initially  identified,  the  model  was  run  with  2 1  replications  in  order  to 
satisfy  the  regression  theory  requirement  for  at  least  as  many  replications  as  potential 
predictors  (McClave  et  al.  2005).  Additionally,  each  potential  control  was  traced  to  a 
particular  function  within  the  FX99  model.  The  controls  and  their  associated  functions 
where  known  are  shown  in  Table  4.1: 


Table  4.1:  List  of  Potential  Controls  and  Associated  Functions 


Control  Parameter  1 

Parameter2 

Function 

1 

0.004167 

0.00125 

End  of  runway  check 

2 

0.007083 

0.002083 

? 

3 

0.008333 

0.0025 

Start  engines 

4 

0.010417 

0.002917 

Load  MBRK 

5 

0.011 666667  0.0029 1 6667  UnMSRK 

6 

0.0125 

0.00375 

Load  Bomb 

7 

0.012917 

0.00375 

Load  missiles 

8 

0.020833 

0.005833 

Jhalon  service 

9 

0.02125 

0.005833 

Jlox  service 

10 

0.025 

0.007083 

Load  chaff  dispenser 

11 

0.033333 

0.009583 

Do  preflight 

12 

0.034167 

0.009583 

Ranks 

13 

0.041667 

0.024167 

refill  lox  cart 

14 

50000.5 

multiple  networks,  multiple  tasks 

15 

50000.5 

16 

50000.5 

17 

50000.5 

18 

50000.5 

19 

50000.5 

20 

50000.5 
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Table  4. 1  lists  the  potential  control  in  the  first  column.  The  second  column,  Parameter  1, 
is  the  expected  mean.  The  column  labeled  Parameter2  is  the  expected  standard  deviation. 
Finally,  the  function  of  each  control,  if  known,  is  listed  in  the  fourth  column.  Controls  1- 
13  appear  to  be  flightline  pre-  and  post-sortie  tasks  and  are  all  lognormally  distributed. 
Controls  14-20  are  random  variates  for  node  selection  in  task  networks  with  multiple 
options  and  a  specific  probability  associated  with  the  task  selection.  Controls  14-20  are 
all  random  variates  drawn  using  the  RANDI.F  function  generating  a  unifonn,  discrete 
random  integer  between  1  and  100000.  The  specific  function  for  Control  2  could  not  be 
identified. 

After  calculating  the  difference  between  the  known  and  expected  value  for  each 
potential  control  following  each  replication  (Appendix  F),  the  values  were  placed  in  the 
20  X  2 1  matrix  shown  in  Appendix  G.  The  20X21  matrix  provided  the  data  for  the 
stepwise  regression,  and  the  data  was  imported  in  Minitab  14  for  the  stepwise  regression 
calculations,  first  using  C15  as  the  response  variable.  An  initial  alpha  level  of  0.05  was 
set,  and  the  stepwise  regression  returned  just  one  control,  the  UnMSRK  task,  which 
involves  removing  the  missile  rack  from  the  aircraft. 

After  performing  the  control  variate  calculations  in  Appendix  A,  the  new,  reduced 
confidence  interval  was  calculated.  The  improvement  was  drastic,  and  can  be  seen  in 
Figure  4.5: 
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Base  —  C15 

95%  Cl:  1.675734014  1.7680755 

Cl  halfwidth:  0.046170748 
Control  Variates  —  C15 
95%  Cl:  1.70881 1625  1.7492334 

Cl  halfwidth:  0.020210863 
Improvement: _ 56.23% _ 

Figure  4.5:  Control  Variates  Results  Summary,  C15,  Single  Control 

Using  the  calculations  in  Appendix  A  to  take  advantage  of  the  correlation  between  the 
UnMSRK  task  and  the  C15  output  statistic,  the  confidence  interval  halfwidth  decreased 
by  56.23  percent.  To  put  this  improvement  into  perspective,  the  number  of  replications 
was  calculated  in  order  to  achieve  the  same  level  of  confidence  with  the  original  model 
using  equation  2. 15  from  Chapter  2.  In  order  to  achieve  the  same  confidence  interval 
halfwidth  as  the  control  variates  model,  the  user  would  need  to  run  approximately  115 
replications  in  order  to  achieve  the  same  confidence  in  the  C15  output  statistic,  versus  21 
replications  with  the  control  variates  model. 

For  the  second  experiment  with  the  response  variable  replaced  by  C24,  the 
stepwise  regression  detennined,  like  the  previous  experiment  with  C 1 5  as  the  predictor, 
that  a  single  control  was  a  predictor  of  the  response  variable,  C24.  In  this  case  the 
predictor  was  control  10,  Load  Chaff  Dispenser.  The  results  from  this  experiment  are 
shown  in  Figure  4.6: 


Base  —  C24 

95%  Cl:  67.5334843  69. 1293728 

Cl  halfwidth:  0.79794424 
Control  Variates  —  C24 
95%  Cl:  68.3349381  68.3421545 

Cl  halfwidth:  0.00360816 
Improvement:  99.55% _ 


Figure  4.6:  Control  Variates  Results  Summary,  C24,  Single  Control 
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The  improvement  in  the  confidence  interval  halfwidth  shown  over  the  original  model 
using  control  10  as  a  predictor  in  the  FX99  model  is  exceptional.  The  results  indicate  an 
extremely  strong  relationship  between  the  load  chaff  dispenser  control  and  the  C24 
output  statistic.  To  achieve  this  same  level  of  confidence  with  the  original  model,  the 
user  would  need  to  make  over  a  million  additional  replications  with  the  original  model. 
This  is  impractical,  given  the  time  constraints  involved  with  the  LCOM  model  runs. 

While  the  results  for  the  control  variates  experiment  show  a  significant 
improvement  in  the  confidence  interval  halfwidth,  the  stepwise  regression  revealed  only 
one  control  with  the  alpha  level  of  0.05.  Since  typical,  real-life  LCOM  models  may 
incorporate  many  more  variables  with  a  much  higher  complexity,  the  possibility  of  a 
particular  model  possessing  multiple  controls  is  great.  In  order  to  demonstrate  the 
technique  using  multiple  controls,  the  stepwise  regression  was  perfonned  a  second  time 
for  the  model,  using  C15  as  the  response  variable,  with  an  alpha  level  of  0. 15.  In  this 
case  the  stepwise  regression  concluded  with  5  controls  identified  as  predictors  of  the 
response  variable,  Cl 5.  The  five  controls  identified  are  listed  in  Figure  4.7: 


Control  5  -  UnMSRK 
Control  10  -  Load  Chaff  Dispenser 
Control  14  -  Task  Selection 
Control  15  -  Task  Selection 
Control  20  -  Task  Selection _ 

Figure  4.7:  List  of  Five  Controls  Identified  With  0.15  Alpha  Level 

The  results  for  this  experiment  were  not  as  favorable  as  the  previous  experiment 
with  just  one  control,  but  still  showed  an  significant  improvement  over  the  original 
confidence  interval  halfwidth.  This  is  expected,  since  the  alpha  level  was  relaxed  to  0.15, 
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making  the  controls  weaker  predictors  of  the  response  variable,  Cl  5.  The  results  for  the 
multiple  control  variate  experiment  are  summarized  in  Figure  4.8: 


Base  —  C15 
95%  Cl: 

1.674593824  1.7692157 

Cl  halfwidth: 

0.047310938 

Control  Variates  —  C15 

95%  Cl: 

1.691849065  1.761883229 

Cl  halfwidth: 

0.035017082 

Improvement: 

25.99% 

Figure  4.8:  Control  Variates  Results  Summary,  C15,  Multiple  Controls 

In  this  case,  the  user  would  be  required  to  make  approximately  63  replications  in  order  to 
achieve  the  same  level  of  confidence  with  the  original  model.  All  calculations  for  the 
control  variates  experiments  were  made  in  Microsoft  Excel.  A  copy  of  the  full  results 
can  be  seen  in  Appendix  J. 
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5.  Discussion 


This  section  contains  all  of  the  conclusions  from  this  research  and 
recommendations  for  further  research  concerning  the  topic  of  variance  reduction  within 
LCOM.  It  contains  conclusions  summarized  from  the  analysis  of  Chapter  4. 
Additionally,  it  concludes  with  recommendations  for  further  research. 

5.1.  Conclusions 

5.1.1.  Common  Random  Numbers 

The  common  random  numbers  experiment  exhibited  no  improvement  when  using 
the  C 1 5  statistic  as  the  output  variable  of  interest,  but  exhibited  limited  improvement 
when  using  the  C24  statistic.  After  investigating  the  configuration  of  random  variate 
draws  in  the  original  model,  the  variance  of  the  C24  output  statistic  was  not  unifonnly 
reduced. 

Recall  that  the  common  random  numbers  theory  relies  on  synchronization  of 
random  variate  draws.  Each  function  within  the  model  would  need  a  separate  random 
variate  draw  that  could  be  synchronized  by  designating  a  unique  random  number  stream 
or  starting  seed.  Furthermore,  in  a  model  such  as  LCOM  where  nearly  every  individual 
task  has  a  significant  impact  on  the  output  variables  of  interest,  the  individual  tasks 
themselves  must  also  be  isolated  so  they  can  each  be  given  a  unique  random  number 
stream  or  starting  seed.  In  the  current  configuration,  LCOM  often  uses  a  single  random 
variate  source  to  generate  input  samples  for  multiple  functions  as  well  as  dozens  or  even 
hundreds  of  tasks.  Consequently,  improvements  observed  using  the  common  random 


46 


numbers  method  cannot  be  guaranteed.  In  this  configuration,  common  random  numbers 
is  not  a  feasible  method  of  variance  reduction. 

5.1.2.  Antithetic  Variates 

The  results  for  both  the  C 1 5  output  statistic  and  the  C24  output  statistic  were 
unfavorable  for  variance  reduction.  In  fact,  just  the  opposite  occurred.  Both  experiments 
concluded  with  an  increase  in  the  variance,  rendering  the  use  of  antithetic  variates 
unsuccessful  in  the  search  for  a  feasible  variance  reduction  method. 

Like  common  random  numbers,  antithetic  variates  relies  on  random  variate 
synchronization  in  order  to  induce  a  correlation.  In  CRN,  this  correlation  occurs  across 
multiple  scenarios.  In  AV,  the  correlation  is  induced  within  replications  of  a  single 
scenario.  The  AV  method  is  unsuccessful  in  LCOM  for  the  same  reason  the  CRN 
method  failed  -  the  inability  to  synchronize  random  variates  in  LCOM  used  for  the  same 
purpose.  Since  several  random  variate  draws  generate  input  for  multiple  purposes  within 
the  model,  the  possibility  for  synchronization  within  the  current  framework  does  not 
exist.  Also,  like  common  random  numbers,  the  results  are  simply  due  to  randomness. 
Consequently,  the  increases  in  the  confidence  intervals  using  the  AV  method  do  not 
necessarily  indicate  a  less  effective  model,  but  simply  an  ineffective  method  of  variance 
reduction. 

5.1.3.  Control  Variates 

Unlike  the  previous  two  methods,  control  variates  capitalizes  on  an  existing 
correlation  between  random  inputs  and  a  particular  output  variable  of  interest.  In  this 
experiment  using  the  FX99  model,  control  variates  performed  extremely  well.  In  all 
cases  using  both  the  C15  output  statistic  and  the  C24  output  statistic,  control  variates 
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produced  a  significant  improvement  in  the  output  variance.  A  significant  reduction  in  the 
output  variance  equates  to  a  more  accurate  estimate  of  the  mean.  Additionally,  with  a 
halfwidth  goal  in  mind,  a  reduction  in  variance  can  reduce  the  amount  of  time  the  user 
spends  perfonning  additional  replications  in  order  to  achieve  that  specified  halfwidth 
goal.  In  one  case,  as  shown  in  the  CV  experiment  using  the  C24  output  statistic,  the 
variance  reduction  achieved  by  the  control  variates  model  was  so  significant  that  over  a 
million  replications  would  be  necessary  to  achieve  the  same  halfwidth  with  the 
unmodified  LCOM  model. 

5.2.  Recommendations  for  Further  Research 

5.2.1.  Improvements  to  LCOM  -  Implementation  of  Control  Variates 
Implementing  a  permanent  option  for  the  control  variates  method  in  LCOM  is 
relatively  simple  for  a  particular  model.  However,  the  controls  may  change  from  model 
to  model.  Therefore,  the  potential  controls  for  each  model  must  be  captured  and  arranged 
in  the  same  manner  as  the  matrix  found  in  Appendix  G,  listing  potential  controls  and  the 
output  variable  of  interest.  This  can  be  performed  fairly  easily  by  a  postprocessor.  It 
would  first  require  the  postprocessor  to  capture  the  random  variates  and  their  known 
expected  value,  and  subtracting  the  observed  mean  from  the  expected  value  to  obtain  the 
difference.  Then,  the  stepwise  regression  must  be  perfonned  in  order  to  identify  the 
actual  predictors  of  the  response  variable.  A  stepwise  regression  could  be  perfonned  by  a 
postprocessor  or  a  statistical  software  package  such  as  Minitab. 

Once  the  controls  are  identified,  the  calculations  in  Appendix  A  must  be 
performed  in  order  to  calculate  the  new,  improved  confidence  interval.  These  steps  can 
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also  be  performed  by  a  postprocessor  -  one  for  a  single  control  and  a  separate 
postprocessor  for  multiple  controls,  since  the  methodology  is  slightly  different. 

5.2.2.  Improvements  to  LCOM  -  Random  Number  Generator 

As  discussed  in  Chapter  2  of  this  research,  the  LCG  used  for  random  number 
generation  in  the  LCOM  model  could  be  considered  inadequate  given  the  relatively  small 
period,  or  stream  length,  and  the  small  number  of  random  number  streams  offered  by  the 
LCG.  With  33  random  variate  draws,  the  modeler  ideally  should  have  at  least  33 
different  streams  of  numbers  in  order  to  dedicate  a  unique  stream  to  each  particular  point 
of  consumption.  In  LCOM,  multiple  tasks  often  sample  from  the  same  random  variate 
draws,  so  ideally  the  tasks  and  functions  would  be  separated  and  isolated.  Then,  a  unique 
stream  or  seed  value  should  be  dedicated  to  each  random  variate  draw  used  for  a 
particular  purpose.  Since  the  LCG  in  SIMSCRIPT  II. 5  only  has  10  options  to  choose 
from,  other  options  for  random  number  generation  should  be  explored. 

Several  alternatives  to  LCGs  exist  in  the  random  number  generation  world.  One 
of  these  alternatives  is  found  in  the  simulation  modeling  software  Arena  Version  Seven 
and  later,  called  a  combined  multiple  recursive  generator.  The  generator  is  titled, 
MRG32k3a  by  L’Ecuyer  (L’Ecuyer  et  al.  2002).  This  generator,  based  on  research  by 
L’Ecuyer,  Simard,  Chen,  and  Kelton,  “differs  in  that  (1)  it  involves  two  separate 
component  generators  that  are  then  combined,  and  (2)  the  recursion  to  get  the  next  values 
looks  back  beyond  just  the  single  preceding  value”  (Kelton  et  al.  2004).  The  cycle 
lengths  are  much  longer  than  LCGs  -  instead  of  burning  the  entire  stream  of  numbers  in  a 
matter  of  minutes,  the  new  generator  will  take  an  ordinary  personal  computer  2.78  x  104<) 
millennia.  Furthermore,  the  number  of  separate  streams  improves  from  10  unique 
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random  number  streams  to  1.8  x  1019  streams  (Kelton  et  al.  2004).  Fortunately,  L’Ecuyer 
and  his  colleagues  have  developed  a  package  that  allows  a  simple  implementation  in 
various  languages  such  as  Java,  C,  and  C++.  An  experienced  programmer  could  easily 
explore  the  implementation  of  L’Ecuyer’s  multiple  recursive  generator  in  LCOM, 
allowing  for  synchronization  of  all  points  of  consumption  for  common  random  numbers. 

5.2.3.  Further  Variance  Reduction  Applications  to  LCOM 

This  research  effort  explored  the  application  of  each  of  the  three  variance 
reduction  techniques  -  common  random  numbers,  antithetic  variates,  and  control  variates 
-  in  isolation.  However,  the  possibility  exists  for  combining  multiple  variance  reduction 
techniques  and  applying  them  to  LCOM  in  order  to  achieve  an  even  greater  level  of 
variance  reduction  observed  with  a  single  technique,  as  was  done  by  Bednar  (2005). 

Furthermore,  the  techniques  were  applied  to  a  single,  hypothetical  model 
developed  for  educational  aid.  While  the  basic  logic  of  the  control  variates  model 
remains  the  same,  the  technique  should  be  applied  to  real-world  models  and  analyzed  for 
similar  effectiveness. 

5.2.4.  Impact  to  the  LCOM  Optimizer 

After  the  techniques  are  implemented  into  real-world  models  with  some  level  of 
improved  variance  reduction,  the  next  question  is,  how  do  the  techniques  affect  the 
performance  of  the  optimizer?  Can  the  control  variates  method  improve  the  perfonnance 
of  the  optimizer?  If  not,  can  the  control  variates  method  be  applied  to  the  optimizer 
itself?  In  theory,  a  successful  variance  reduction  application  to  LCOM  would  mean  a 
more  effective,  predictable  optimizer  performance.  However,  the  actual  effectiveness 
should  be  tested  and  analyzed  to  measure  the  improvement,  if  any  exists. 
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Appendix  A:  Control  Variates  Derivation 


The  following  derivation  is  quoted  directly  from  Bednar’s  thesis,  Feasibility  of  Variance 
Reduction  in  the  Thunder  Campaign  Model  (2005): 

Consider  the  case  were  there  is  only  a  single  control.  Now,  assume  there  is  a 
mean  response  of  interest  from  the  simulation  called  /uY  for  which  7  is  an  estimator. 

Also,  assume  there  is  another  output  variable,  X,  that  is  correlated  with  the  Y  response 
and  has  an  expected  value  //  x  that  is  known.  Since  X  is  correlated  with  the  7  variable,  it 
is  known  as  the  control  variable.  Now  consider  the  controlled  estimator  Y(b)  given  in 
equation  ((A.l)A.l)  where  b  is  a  constant. 

Y(b)=Y-b(X-nx)  (A1) 

Note  that  Y(b)  is  an  unbiased  estimator  of  /jy  by  equation  (A.2). 

E[Y(b)]=E[Y]-E[b(X-vs)]  (A.2) 

E[Y{b)\=M, -b(ux -Ms)  =  Mr  I  A.  3) 

The  variance  of  Y(b)  is  given  in  equation  (A. 4). 

Var(y  (6))  =  Var(T)  +  b2 Var(x)  -  2hCov(7,  x)  (A.4) 

With  a  little  manipulation  of  equation  (A.4)  it  can  be  shown  that  the  variance  of  Y(b)  is 
smaller  than  the  variance  of  Y  if  equation  (A. 5)  holds. 

2hCov(7,X)>/rVar(x)  (A.5) 

In  observing  equation  (A.5),  it  is  apparent  that  if  the  variables  X  and  7  are  independent, 
then  the  Cov(A,  7)  =  0  and  it  follows  that  there  can  be  no  reduction  in  variance  of  7. 
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To  find  the  value  of  b  that  minimizes  equation  (A.4)  the  derivative  with  respect  to 


b  is  found. 


dVar(y(/>))  =  2b\ax(x)  -  2Cov(7,  x) 
8b 


(A.  6) 


From  equation  (A. 6)  the  minimum  point  is  found  by  setting  the  derivative  to  zero. 


Equation  (A. 7)  is  the  candidate  for  b  that  minimizes  equation  (A.4)(A.4)  thus  labeled  (3. 


a  Cov(r,z) 
P  Var(x) 


(A.  7) 


To  verify  this  is  a  minimum,  the  second  derivative  is  found. 


a2Var m  =  2Mx) 


(A.8) 


Since  the  variance  is  always  non-negative,  then  equation  (A. 7)  is  the  value  for  b  that 
minimizes  equation  (A.4).  Combining  equation  (A.4),  equation  (A. 7),  and  using  some 


simple  algebra  yields  equation  (A. 9). 


V ar(r(/?))  =  (l  -  )v ar(Y) 


(A. 9) 


Where  pXY  is  the  correlation  coefficient  between  X  and  Y.  Var(Y(/:/))  is  the  minimum 
variance.  The  controlled  observations  (A.  10)  are  averaged  (A.l  1)  to  obtain  an  unbiased 
estimator  of  /jy  • 


Yi{j3)  =  Yi-j3{Xi-Jux),i  =  l,...,K 


(A.  10) 


1=1 


(A.  11) 


where  K  is  the  sample  size. 
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Since  the  value  of  / 3  is  unknown,  it  must  be  estimated.  An  estimate  of  / 3  can  be 
found  by  substituting  the  sample  quantities  into  equation  (A. 7)  This  solution  is  the  least 
squares  solution  for  fi  The  least  squares  solution  is  also  the  maximum  likelihood 
solution  with  the  assumption  of  joint  nonnality  between  X  and  7.  Equation  (A.  12) 
estimates  ft. 


(A.  12) 


The  point  estimator  of  jUY  is  estimated  by  equation  (A.  13). 

*(/»)=  (A.  13) 

A  I=i 

Using  regression  theory  an  interval  estimate  for  /jy  can  be  obtained.  By  making 
the  assumption  of  joint  normality  for  X  and  Y,  the  conditional  distribution  of  Y  given  X 
will  be  normal  by  equation  (A.  14) 


7|X  =  x~n(//},  +/?(*-//  x),cx2E) 


(A.  14) 


where 


(A.  15) 


and 

=  Var(7)  (A.  16) 

Since  the  values  of  the  control  variable  X  and  it  mean  /jx  are  known,  then  it  can  be  seen 
that  the  conditional  mean  of  7  given  X has  two  terms.  The  terms  are  broken  into  the 
parameter  to  be  estimated  /jy  and  a  correction  term.  To  get  the  juY  term,  the  corrections 
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need  to  be  subtracted  out  as  in  equation  (A.  10).  From  equation  (A.  14),  equation  (A.  17) 
can  be  formed. 


Yi  =jUY  +  /3(Xi-Jux)+£i,l<i<K  (A.  17) 

where  si  are  the  residuals  and  are  of  the  form  in  equation  (A.  18). 

A~n(o,^2)  (A.  18) 

Since  the  values  of  /jy  and  J3  are  unknown,  the  method  of  least  squares  can  be  applied  to 
solve  for  them.  /jy  will  be  the  intercept  and  is  normally  distributed  as  in  equation 
(A.  19). 

My  (fi\  ~  Hmy  >  v2esi  1 )  (A. 1 9) 

The  value  of  sn  in  equation  (A.  19)  is  the  upper  left  hand  entry  in  the  matrix  (dtd)  ' 

where  D  is  of  the  form  in  equation  (A.20). 

1  Xl  —  fdx 
1  A2  ~  Mx 

D=  1  X3~jux  (A.20) 

1  X  K  —  fJ.x  _ 

To  generate  a  confidence  interval  about  juY[fi),  cr]  must  be  estimated.  Since  a2 
represents  the  variability  in  Y  given  X,  the  formula  for  the  residual  mean  square  error  is 
used. 


where 


54 


Y\B  = 


By{fj)+  p{xi  -jUx),\<i 


<i  <K 


(A.22) 


It  can  be  seen,  from  the  above  equations,  that 


MP  ~By 


1  lK- 2 


<j2s  2 
0  All 

K-2 


(A.23) 


has  a  Student-t  distribution  with  K-2  degrees  of  freedom.  Therefore,  the  confidence 
interval  for  /jy  is  given  by 


My  ±t 


aW^H 


(A.24) 


Now  in  simulations  there  are  possibly  more  than  one  control  for  a  response. 
Therefore,  equation  (A.  17)  is  modified  to  be 


r,  =  +  -Ux)+e„l 


(A.25) 


where, 


Q<K- 1 


(A.26) 


Therefore,  equation  (A.21)  is  reformed  into 


K  (  f  Q  ^  ,  \ 

X  Yi-  My+'EPj \xj~BXj) 


K-Q-l 


(A.27) 


and  ju  is  the  upper  left  hand  entry  in  the  matrix  (d  t  d  )  '  where  D  is  of  the  form 
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'1 

X\  j  filx 

X2l  _  A 2X 

XQ\  ~  Mqx 

1 

A 

1 

cn 

X2l  ~  f^2X 

Xqi  ~  Mqx 

D  = 

1 

X13  A IX 

X2\  ~  Mix 

Xq\  —  Mqx 

1 

A 

•  1 

* 

X2i  ~  Mix 

1 
•  1 

>3) 

(A.28) 


Then  the  1 00(  \-a)%  confidence  interval  in  equation  (A.24)  becomes 


/uY+t 


K-Q-H  '-j 


(A.29) 
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Appendix  B:  Calculations,  Paired  Difference  Test  of  Hypothesis  for  pd  =  (pi  -  p2) 
Using  FX-99  Scenarios  with  Unconstrained  and  Constrained  Manpower 
C15  (Overall  Sorties  per  Aircraft  per  Day): 


Replication 

unconstrained 

constrained 

difference 

1 

2.02 

1.95 

0.07 

2 

2.16 

1.98 

0.18 

3 

2.26 

1.92 

0.34 

4 

2.15 

2 

0.15 

5 

2.25 

2.02 

0.23 

6 

2.25 

1.93 

0.32 

7 

2.33 

1.99 

0.34 

8 

2.13 

1.98 

0.15 

9 

2.11 

1.94 

0.17 

10 

2.33 

2.04 

0.29 

11 

2.12 

1.98 

0.14 

12 

2.14 

1.88 

0.26 

13 

2.15 

1.99 

0.16 

14 

2.22 

1.88 

0.34 

15 

2.18 

1.92 

0.26 

16 

2.18 

1.95 

0.23 

17 

2.27 

2.04 

0.23 

18 

2.4 

2.04 

0.36 

19 

2.17 

1.94 

0.23 

20 

2.13 

1.9 

0.23 

21 

2.2 

1.99 

0.21 

22 

2.17 

1.95 

0.22 

23 

2.25 

2 

0.25 

24 

2.17 

1.86 

0.31 

25 

2.15 

1.94 

0.21 

26 

2.29 

2.01 

0.28 

27 

2.25 

1.96 

0.29 

28 

2.2 

1.95 

0.25 

29 

2.27 

1.99 

0.28 

30 

2.18 

1.97 

0.21 

Mean 

2.202666667 

1.963 

0.23966667 

St  Dev 

0.07851788 

0.04742689 

0.06910678 

t 

18.9953632 

95%  Cl: 

0.214937123 

0.26439621 

Absolute  value  of  t  (19.32)  is  greater  than  to.025  (1-96),  therefore  we  can  reject  the  null 
hypothesis  that  the  two  population  means  are  equal.  Furthermore,  the  confidence  interval 
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does  not  include  zero.  We  can  infer  that  the  mean  for  the  unconstrained  scenario  exceeds 


the  mean  for  the  constrained  scenario. 


C24  (Mission  Capable  Rate): 


Replication 

unconstrained 

constrained 

difference 

1 

68.26 

61.74 

6.52 

2 

68.81 

61.66 

7.15 

3 

66.37 

65.16 

1.21 

4 

67.19 

61.81 

5.38 

5 

67.01 

61.61 

5.4 

6 

66.74 

65.04 

1.7 

7 

65.29 

60.77 

4.52 

8 

68.62 

61.37 

7.25 

9 

67.35 

61.74 

5.61 

10 

65.71 

62.8 

2.91 

11 

67.28 

62.33 

4.95 

12 

66.96 

65.11 

1.85 

13 

67.35 

63.11 

4.24 

14 

65.72 

64.07 

1.65 

15 

67.61 

65.92 

1.69 

16 

67.17 

61.87 

5.3 

17 

66.29 

62.6 

3.69 

18 

64.67 

62.8 

1.87 

19 

66.54 

65.45 

1.09 

20 

67.26 

59.93 

7.33 

21 

66.95 

60.74 

6.21 

22 

67.54 

61.44 

6.1 

23 

66.85 

61.86 

4.99 

24 

67.28 

64.29 

2.99 

25 

67.18 

63.03 

4.15 

26 

66.43 

61.74 

4.69 

27 

66.97 

61.8 

5.17 

28 

66.4 

65.3 

1.1 

29 

66.6 

62.45 

4.15 

30 

66.79 

65.74 

1.05 

Mean 

66.90633333 

62.8426667 

4.06366667 

St  Dev 

0.875090077 

1.68354702 

2.03690855 

t 

10.9271567 

95%  Cl: 

3.334768304 

4.79256503 

Absolute  value  of  t  ( 1 1 . 1 1 )  is  greater  than  to.025  ( 1  -96),  therefore  we  can  reject  the  null 
hypothesis  that  the  two  population  means  are  equal.  Furthermore,  the  confidence  interval 
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does  not  include  zero.  We  can  infer  that  the  mean  for  the  unconstrained  scenario  exceeds 
the  mean  for  the  constrained  scenario. 


59 


Appendix  C: 

Merged  Ouput  Report  Example 

for  FX99  Model, 

Statistics  CIO 

-C25 

kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk 

•k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k 

kk-kk-k-kk-k 

kkkkkkkkkkk-* 

rkkkkkkkkkkkkkkkk 

ASC  LCOM  2 . 

8 

PERIOD  1  FROM  0.  TO  1.000 

LEVEL  1  REPORT  - 

NOT  CONDENSED 

95%  CONFIDENCE 

INTERVAL 

STATISTICS  FROM  10  REPLICATIONS  MERGED. 

AVERAGE  STD 

DEV 

MINIMUM 

MAXIMUM 

LOWER 

UPPER 

kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk 

kkkkkkkkkkkkkkkkkk 

kkkkkkkk 

kkkkkkkkkkkkkkkkk 

CIO  AVG.  AC  POST  SORTIE  TIME (HRS) 

1.000  OVERALL 

.880 

.02 

.  844 

.907 

.865 

.894 

CIO  AVG.  AC  POST  SORTIE  TIME (HRS) 

1.000  FX-99 

.880 

.02 

.  844 

.907 

.865 

.894 

Cll  MIN.  AC  POST  SORTIE  TIME (HRS) 

1.000  OVERALL 

.341 

.03 

.302 

.397 

.324 

.357 

Cll  MIN.  AC  POST  SORTIE  TIME (HRS) 

1.000  FX-99 

.341 

.03 

.302 

.397 

.324 

.357 

C12  MAX.  AC  POST  SORTIE  TIME (HRS) 

1.000  OVERALL 

1.893 

.18 

1 . 676 

2.233 

1.780 

2.006 

C12  MAX.  AC  POST  SORTIE  TIME (HRS) 

1.000  FX-99 

1.893 

.18 

1 . 676 

2.233 

1.780 

2.006 

C13  STD  DEV  POST  SORTIE  TIME (HRS) 

1.000  OVERALL 

.350 

.02 

.309 

.399 

.335 

.365 

C13  STD  DEV  POST  SORTIE  TIME (HRS) 

1.000  FX-99 

.350 

.02 

.309 

.399 

.335 

.365 

C14  REQUESTED  SORTIES/  AC  /DAY 

1.000  OVERALL 

3.583 

0. 

3.583 

3.583 

N/A 

N/A 

C14  REQUESTED  SORTIES/  AC  /DAY 

1.000  FX-99 

3.583 

0. 

3.583 

3.583 

N/A 

N/A 

Cl 5  ACHIEVED  SORTIES/  AC  /DAY 

1.000  OVERALL 

2.425 

.12 

2.229 

2.625 

2.352 

2.498 

C15  ACHIEVED  SORTIES/  AC  /DAY 

1.000  FX-99 

2.425 

.12 

2.229 

2.625 

2.352 

2.498 

Cl 6  FLYING  HOURS 

1.000  OVERALL 

204.011  7 

.56 

193.644 

216.647 

199.326 

208.695 

Cl 6  FLYING  HOURS 

1.000  FX-99 

204.011  7 

.56 

193.644 

216.647 

199.326 

208.695 

C17  AVG.  FLYING  HOURS  /  AC  /  DAY 

1.000  OVERALL 

4.250 

.16 

4.034 

4.513 

4.153 

4.348 

C17  AVG.  FLYING  HOURS  /  AC  /  DAY 

1.000  FX-99 

4.250 

.16 

4.034 

4.513 

4.153 

4.348 

Cl 8  AVG.  AC  PRE  SORTIE  TIME  (HRS) 

1.000  OVERALL 

1.966 

.08 

1.855 

2.122 

1.918 

2.014 

Cl 8  AVG.  AC  PRE  SORTIE  TIME  (HRS) 

1.000  FX-99 

1 . 966 

.08 

1.855 

2.122 

1.918 

2.014 

Cl 9  MIN.  AC  PRE  SORTIE  TIME  (HRS) 

1.000  OVERALL 

.333 

.02 

.291 

.355 

.319 

.348 

Cl 9  MIN.  AC  PRE  SORTIE  TIME  (HRS) 

1.000  FX-99 

.333 

.02 

.291 

.355 

.319 

.348 

C20  MAX.  AC  PRE  SORTIE  TIME  (HRS) 

1.000  OVERALL 

5.043 

.52 

4 .141 

5.727 

4.722 

5.365 

C20  MAX.  AC  PRE  SORTIE  TIME  (HRS) 

1.000  FX-99 

5.043 

.52 

4 .141 

5.727 

4.722 

5.365 

C21  STD  DEV  PRE  SORTIE  TIME  (HRS) 

1.000  OVERALL 

1.071 

.08 

.  925 

1.202 

1.022 

1.120 

C21  STD  DEV  PRE  SORTIE  TIME  (HRS) 

1.000  FX-99 

1.071 

.08 

.  925 

1.202 

1.022 

1.120 

C22  SGR  REQUIREMENT  -  AS  INPUT 

1.000  OVERALL 

0. 

0. 

0. 

0. 

N/A 

N/A 

C22  SGR  REQUIREMENT  -  AS  INPUT 

1.000  FX-99 

0. 

0. 

0. 

0. 

N/A 

N/A 

C23  SGR  REQUIREMENT  -  %  REALIZED 

1.000  OVERALL 

0. 

0. 

0. 

0. 

N/A 

N/A 

C23  SGR  REQUIREMENT  -  %  REALIZED 

1.000  FX-99 

0. 

0. 

0. 

0. 

N/A 

N/A 

C24  %MISSION  CAPABLE/MC  RATE 

1.000  OVERALL 

61 .441 

.71 

60.299 

62.393 

61.004 

61.878 

C24  %MISSION  CAPABLE/MC  RATE 

1.000  FX-99 

61 . 441 

.71 

60.299 

62.393 

61.004 

61.878 

C25  %NOT  MISSION  CAPABLE/NMC  RATE 

1.000  OVERALL 

38.559 

.71 

37 . 607 

39.701 

38.122 

38.996 

C25  %NOT  MISSION  CAPABLE/NMC  RATE 

1.000  FX-99 

38.559 

.71 

37 . 607 

39.701 

38.122 

38.996 
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Appendix  D:  Screen  Shot,  ISEEDS  Tab  in  LCOM  Graphical  User  Interface 


ASC  LCOM  2.8 
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Appendix  E:  Screen  Shot,  Production  Run  Settings  in  LCOM  Graphical  User 


Interface 


ASC  LCOM  2.8 
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Appendix  F:  Potential  Control  Observations  for  21  Replications 


Rep  1 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004296355 

-0.000129355 

2 

0.007083 

0.002083 

0.006909073 

0.000173927 

3 

0.008333 

0.0025 

0.008225659 

0.000107341 

4 

0.010417 

0.002917 

0.01075457 

-0.00033757 

5 

0.01166667 

0.002916667 

0.012967107 

-0.00130044 

6 

0.0125 

0.00375 

0.011788496 

0.000711504 

7 

0.012917 

0.00375 

0.013392901 

-0.000475901 

8 

0.020833 

0.005833 

0.020451375 

0.000381625 

9 

0.02125 

0.005833 

0.022362195 

-0.001112195 

10 

0.025 

0.007083 

0.02581281 

-0.00081281 

11 

0.033333 

0.009583 

0.033729431 

-0.000396431 

12 

0.034167 

0.009583 

0.032269908 

0.001897092 

13 

0.041667 

0.024167 

0.037603729 

0.004063271 

14 

50000.5 

3000 

52359.79464 

-2359.294643 

15 

50000.5 

6000 

48885.21818 

1115.281818 

16 

50000.5 

10000 

47602.13793 

2398.362069 

17 

50000.5 

15000 

48855.50495 

1144.99505 

18 

50000.5 

50000 

47690.16477 

2310.335227 

19 

50000.5 

75000 

60801.71429 

-10801.21429 

20 

50000.5 

1-100000 

49346.29717 

654.2028302 

C15 

1.72 

C24 

67.91 
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Rep  2 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004120445 

4.65552E-05 

2 

0.007083 

0.002083 

0.006814739 

0.000268261 

3 

0.008333 

0.0025 

0.008510791 

-0.000177791 

4 

0.010417 

0.002917 

0.01090243 

-0.00048543 

5 

0.011666667 

0.002916667 

0.011171826 

0.000494841 

6 

0.0125 

0.00375 

0.012388712 

0.000111288 

7 

0.012917 

0.00375 

0.013102025 

-0.000185025 

8 

0.020833 

0.005833 

0.021410344 

-0.000577344 

9 

0.02125 

0.005833 

0.022341171 

-0.001091171 

10 

0.025 

0.007083 

0.025271086 

-0.000271086 

11 

0.033333 

0.009583 

0.03328961 

4.33904E-05 

12 

0.034167 

0.009583 

0.034670938 

-0.000503938 

13 

0.041667 

0.024167 

0.04613256 

-0.00446556 

14 

50000.5 

3000 

52279.18584 

-2278.685841 

15 

50000.5 

6000 

48679.63964 

1320.86036 

16 

50000.5 

10000 

47817.52222 

2182.977778 

17 

50000.5 

15000 

53965.19588 

-3964.695876 

18 

50000.5 

50000 

49043.92571 

956.5742857 

19 

50000.5 

75000 

43168.78571 

6831.714286 

20 

50000.5 

1-100000 

47471.7734 

2528.726601 

C15 

1.68 

C24 

68.76 
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Rep  3 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004307853 

-0.000140853 

2 

0.007083 

0.002083 

0.007103736 

-2.07365E-05 

3 

0.008333 

0.0025 

0.008965121 

-0.000632121 

4 

0.010417 

0.002917 

0.010433697 

-1.66973E-05 

5 

0.011666667 

0.002916667 

0.011003801 

0.000662866 

6 

0.0125 

0.00375 

0.012513773 

-1.37734E-05 

7 

0.012917 

0.00375 

0.012469548 

0.000447452 

8 

0.020833 

0.005833 

0.021016314 

-0.000183314 

9 

0.02125 

0.005833 

0.02087912 

0.00037088 

10 

0.025 

0.007083 

0.026763861 

-0.001763861 

11 

0.033333 

0.009583 

0.035507132 

-0.002174132 

12 

0.034167 

0.009583 

0.03265749 

0.00150951 

13 

0.041667 

0.024167 

0.045782504 

-0.004115504 

14 

50000.5 

3000 

48001.08257 

1999.417431 

15 

50000.5 

6000 

47154.41121 

2846.088785 

16 

50000.5 

10000 

50387.76 

-387.26 

17 

50000.5 

15000 

47536.63918 

2463.860825 

18 

50000.5 

50000 

48828.50811 

1171.991892 

19 

50000.5 

75000 

29984.66667 

20015.83333 

20 

50000.5 

1-100000 

49497.46009 

503.0399061 

C15 

1.72 

C24 

68.14 

65 


Rep  4 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.003979227 

0.000187773 

2 

0.007083 

0.002083 

0.007268175 

-0.000185175 

3 

0.008333 

0.0025 

0.008385633 

-5.26327E-05 

4 

0.010417 

0.002917 

0.010273211 

0.000143789 

5 

0.011666667 

0.002916667 

0.012033733 

-0.000367067 

6 

0.0125 

0.00375 

0.012810408 

-0.000310408 

7 

0.012917 

0.00375 

0.013645831 

-0.000728831 

8 

0.020833 

0.005833 

0.022215128 

-0.001382128 

9 

0.02125 

0.005833 

0.020520842 

0.000729158 

10 

0.025 

0.007083 

0.026348992 

-0.001348992 

11 

0.033333 

0.009583 

0.030584289 

0.002748711 

12 

0.034167 

0.009583 

0.033665912 

0.000501088 

13 

0.041667 

0.024167 

0.048698307 

-0.007031307 

14 

50000.5 

3000 

46744.13402 

3256.365979 

15 

50000.5 

6000 

53270.41414 

-3269.914141 

16 

50000.5 

10000 

53715.96512 

-3715.465116 

17 

50000.5 

15000 

52111.55556 

-2111.055556 

18 

50000.5 

50000 

50603.66883 

-603.1688312 

19 

50000.5 

75000 

49974.5 

26 

20 

50000.5 

1-100000 

49098.07104 

902.4289617 

C15 

1.56 

C24 

69.53 

66 


Rep  5 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004148483 

1.85175E-05 

2 

0.007083 

0.002083 

0.007104216 

-2.12163E-05 

3 

0.008333 

0.0025 

0.00822961 

0.00010339 

4 

0.010417 

0.002917 

0.010479389 

-6.2389E-05 

5 

0.011666667 

0.002916667 

0.010829413 

0.000837254 

6 

0.0125 

0.00375 

0.012823736 

-0.000323736 

7 

0.012917 

0.00375 

0.012600527 

0.000316473 

8 

0.020833 

0.005833 

0.02116291 

-0.00032991 

9 

0.02125 

0.005833 

0.021548201 

-0.000298201 

10 

0.025 

0.007083 

0.025242127 

-0.000242127 

11 

0.033333 

0.009583 

0.035697918 

-0.002364918 

12 

0.034167 

0.009583 

0.034171521 

-4.52103E-06 

13 

0.041667 

0.024167 

0.050881655 

-0.009214655 

14 

50000.5 

3000 

46250.58763 

3749.912371 

15 

50000.5 

6000 

52246.03125 

-2245.53125 

16 

50000.5 

10000 

51623.9881 

-1623.488095 

17 

50000.5 

15000 

49831.4 

169.1 

18 

50000.5 

50000 

50134.54717 

-134.0471698 

19 

50000.5 

75000 

34365.875 

15634.625 

20 

50000.5 

1-100000 

52180.53591 

-2180.035912 

C15 

1.48 

C24 

70.55 

67 


Rep  6 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004131596 

3.54039E-05 

2 

0.007083 

0.002083 

0.00727978 

-0.00019678 

3 

0.008333 

0.0025 

0.008089615 

0.000243385 

4 

0.010417 

0.002917 

0.009961899 

0.000455101 

5 

0.011666667 

0.002916667 

0.01210644 

-0.000439773 

6 

0.0125 

0.00375 

0.012532255 

-3.22554E-05 

7 

0.012917 

0.00375 

0.013623751 

-0.000706751 

8 

0.020833 

0.005833 

0.021688133 

-0.000855133 

9 

0.02125 

0.005833 

0.021222368 

2.76322E-05 

10 

0.025 

0.007083 

0.025193072 

-0.000193072 

11 

0.033333 

0.009583 

0.036221284 

-0.002888284 

12 

0.034167 

0.009583 

0.034567143 

-0.000400143 

13 

0.041667 

0.024167 

0.041411234 

0.000255766 

14 

50000.5 

3000 

53744.81739 

-3744.317391 

15 

50000.5 

6000 

50070.44144 

.69.94144144 

16 

50000.5 

10000 

43754.30682 

6246.193182 

17 

50000.5 

15000 

47674.89 

2325.61 

18 

50000.5 

50000 

48362.47619 

1638.02381 

19 

50000.5 

75000 

56075.38462 

-6074.884615 

20 

50000.5 

1-100000 

49205.52055 

794.9794521 

C15 

1.84 

C24 

67.34 

68 


Rep  7 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004204707 

-3.77073E-05 

2 

0.007083 

0.002083 

0.007334992 

-0.000251992 

3 

0.008333 

0.0025 

0.008138469 

0.000194531 

4 

0.010417 

0.002917 

0.010279924 

0.000137076 

5 

0.011666667 

0.002916667 

0.012402958 

-0.000736292 

6 

0.0125 

0.00375 

0.012501427 

-1.42668E-06 

7 

0.012917 

0.00375 

0.013235647 

-0.000318647 

8 

0.020833 

0.005833 

0.021512591 

-0.000679591 

9 

0.02125 

0.005833 

0.01993951 

0.00131049 

10 

0.025 

0.007083 

0.026737878 

-0.001737878 

11 

0.033333 

0.009583 

0.031797786 

0.001535214 

12 

0.034167 

0.009583 

0.030579392 

0.003587608 

13 

0.041667 

0.024167 

0.04711796 

-0.00545096 

14 

50000.5 

3000 

52818.69492 

-2818.194915 

15 

50000.5 

6000 

52280.73214 

-2280.232143 

16 

50000.5 

10000 

47300.19192 

2700.308081 

17 

50000.5 

15000 

50820.34653 

-819.8465347 

18 

50000.5 

50000 

52517.81977 

-2517.319767 

19 

50000.5 

75000 

65022.14286 

-15021.64286 

20 

50000.5 

1-100000 

55130.81991 

-5130.319905 

C15 

1.74 

C24 

66.6 

69 


Rep  8 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004200342 

-3.33421E-05 

2 

0.007083 

0.002083 

0.007367513 

-0.000284513 

3 

0.008333 

0.0025 

0.008488737 

-0.000155737 

4 

0.010417 

0.002917 

0.01043318 

-1.61797E-05 

5 

0.011666667 

0.002916667 

0.011489171 

0.000177495 

6 

0.0125 

0.00375 

0.012701583 

-0.000201583 

7 

0.012917 

0.00375 

0.01328106 

-0.00036406 

8 

0.020833 

0.005833 

0.020885555 

-5.25546E-05 

9 

0.02125 

0.005833 

0.020163566 

0.001086434 

10 

0.025 

0.007083 

0.024736296 

0.000263704 

11 

0.033333 

0.009583 

0.033304055 

2.89449E-05 

12 

0.034167 

0.009583 

0.034572705 

-0.000405705 

13 

0.041667 

0.024167 

0.04892079 

-0.00725379 

14 

50000.5 

3000 

49241.79817 

758.7018349 

15 

50000.5 

6000 

54307.77982 

-4307.279817 

16 

50000.5 

10000 

54024.27586 

-4023.775862 

17 

50000.5 

15000 

54001.2268 

-4000.726804 

18 

50000.5 

50000 

50491.30128 

-490.8012821 

19 

50000.5 

75000 

49225.45455 

775.0454545 

20 

50000.5 

1-100000 

48511.82524 

1488.674757 

C15 

1.7 

C24 

68.29 

70 


Rep  9 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004167309 

-3.09442E-07 

2 

0.007083 

0.002083 

0.006846198 

0.000236802 

3 

0.008333 

0.0025 

0.008226106 

0.000106894 

4 

0.010417 

0.002917 

0.011494027 

-0.001077027 

5 

0.011666667 

0.002916667 

0.011319871 

0.000346796 

6 

0.0125 

0.00375 

0.011784102 

0.000715898 

7 

0.012917 

0.00375 

0.012450817 

0.000466183 

8 

0.020833 

0.005833 

0.019558048 

0.001274952 

9 

0.02125 

0.005833 

0.022868411 

-0.001618411 

10 

0.025 

0.007083 

0.02493219 

6.78101E-05 

11 

0.033333 

0.009583 

0.032834752 

0.000498248 

12 

0.034167 

0.009583 

0.035936101 

-0.001769101 

13 

0.041667 

0.024167 

0.04255218 

-0.00088518 

14 

50000.5 

3000 

51542.19048 

-1541.690476 

15 

50000.5 

6000 

52593.58163 

-2593.081633 

16 

50000.5 

10000 

50000.375 

0.125 

17 

50000.5 

15000 

50313.12766 

-312.6276596 

18 

50000.5 

50000 

47742.0226 

2258.477401 

19 

50000.5 

75000 

51739.54545 

-1739.045455 

20 

50000.5 

1-100000 

47155.51759 

2844.982412 

C15 

1.62 

C24 

69.86 

71 


Rep  10 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004108175 

5.88252E-05 

2 

0.007083 

0.002083 

0.007200016 

-0.000117016 

3 

0.008333 

0.0025 

0.008449467 

-0.000116467 

4 

0.010417 

0.002917 

0.010458388 

-4.13875E-05 

5 

0.011666667 

0.002916667 

0.011959014 

-0.000292347 

6 

0.0125 

0.00375 

0.012591962 

-9.19621E-05 

7 

0.012917 

0.00375 

0.011951391 

0.000965609 

8 

0.020833 

0.005833 

0.020935622 

-0.000102622 

9 

0.02125 

0.005833 

0.02411771 

-0.00286771 

10 

0.025 

0.007083 

0.025525174 

-0.000525174 

11 

0.033333 

0.009583 

0.028542686 

0.004790314 

12 

0.034167 

0.009583 

0.033711771 

0.000455229 

13 

0.041667 

0.024167 

0.038586242 

0.003080758 

14 

50000.5 

3000 

51209.83186 

-1209.331858 

15 

50000.5 

6000 

49723.68182 

276.8181818 

16 

50000.5 

10000 

55195.87912 

-5195.379121 

17 

50000.5 

15000 

46435.88462 

3564.615385 

18 

50000.5 

50000 

47370.86364 

2629.636364 

19 

50000.5 

75000 

61637.7 

-11637.2 

20 

50000.5 

1-100000 

51692.85648 

-1692.356481 

C15 

1.84 

C24 

66.97 

72 


Rep  1 1 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004112574 

5.4426E-05 

2 

0.007083 

0.002083 

0.007053084 

2.99162E-05 

3 

0.008333 

0.0025 

0.00874583 

-0.00041283 

4 

0.010417 

0.002917 

0.010656149 

-0.000239149 

5 

0.011666667 

0.002916667 

0.011673014 

-6.3468E-06 

6 

0.0125 

0.00375 

0.012950998 

-0.000450998 

7 

0.012917 

0.00375 

0.01276583 

0.00015117 

8 

0.020833 

0.005833 

0.022030005 

-0.001197005 

9 

0.02125 

0.005833 

0.023138913 

-0.001888913 

10 

0.025 

0.007083 

0.026502071 

-0.001502071 

11 

0.033333 

0.009583 

0.031960377 

0.001372623 

12 

0.034167 

0.009583 

0.035208701 

-0.001041701 

13 

0.041667 

0.024167 

0.056240337 

-0.014573337 

14 

50000.5 

3000 

47173.23894 

2827.261062 

15 

50000.5 

6000 

48430.53211 

1569.96789 

16 

50000.5 

10000 

49747.77895 

252.7210526 

17 

50000.5 

15000 

43982.77451 

6017.72549 

18 

50000.5 

50000 

50893.68539 

-893.1853933 

19 

50000.5 

75000 

52407.66667 

-2407.166667 

20 

50000.5 

1-100000 

45005.93458 

4994.565421 

C15 

1.76 

C24 

66.78 

73 


Rep  12 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004035946 

0.000131054 

2 

0.007083 

0.002083 

0.006923183 

0.000159817 

3 

0.008333 

0.0025 

0.00817222 

0.00016078 

4 

0.010417 

0.002917 

0.010426167 

-9.16679E-06 

5 

0.011666667 

0.002916667 

0.011815179 

-0.000148513 

6 

0.0125 

0.00375 

0.013616794 

-0.001116794 

7 

0.012917 

0.00375 

0.013356403 

-0.000439403 

8 

0.020833 

0.005833 

0.021742111 

-0.000909111 

9 

0.02125 

0.005833 

0.021551092 

-0.000301092 

10 

0.025 

0.007083 

0.025142272 

-0.000142272 

11 

0.033333 

0.009583 

0.034847684 

-0.001514684 

12 

0.034167 

0.009583 

0.036506547 

-0.002339547 

13 

0.041667 

0.024167 

0.048573483 

-0.006906483 

14 

50000.5 

3000 

50306.52041 

-306.0204082 

15 

50000.5 

6000 

52495 

-2494.5 

16 

50000.5 

10000 

49475.1828 

525.3172043 

17 

50000.5 

15000 

45055.12222 

4945.377778 

18 

50000.5 

50000 

51336.19608 

-1335.696078 

19 

50000.5 

75000 

75419.75 

-25419.25 

20 

50000.5 

1-100000 

48007.5508 

1992.949198 

C15 

1.6 

C24 

68.97 

74 


Rep  13 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004199691 

-3.26909E-05 

2 

0.007083 

0.002083 

0.007141175 

-5.81748E-05 

3 

0.008333 

0.0025 

0.008143035 

0.000189965 

4 

0.010417 

0.002917 

0.009802972 

0.000614028 

5 

0.011666667 

0.002916667 

0.011230763 

0.000435903 

6 

0.0125 

0.00375 

0.011092354 

0.001407646 

7 

0.012917 

0.00375 

0.012545133 

0.000371867 

8 

0.020833 

0.005833 

0.020301339 

0.000531661 

9 

0.02125 

0.005833 

0.021801471 

-0.000551471 

10 

0.025 

0.007083 

0.02480466 

0.00019534 

11 

0.033333 

0.009583 

0.034838308 

-0.001505308 

12 

0.034167 

0.009583 

0.033355928 

0.000811072 

13 

0.041667 

0.024167 

0.036094959 

0.005572041 

14 

50000.5 

3000 

53352.03571 

-3351.535714 

15 

50000.5 

6000 

50090.36449 

-89.86448598 

16 

50000.5 

10000 

45972.11111 

4028.388889 

17 

50000.5 

15000 

48662.45745 

1338.042553 

18 

50000.5 

50000 

47776.87429 

2223.625714 

19 

50000.5 

75000 

61490.54545 

-11490.04545 

20 

50000.5 

1-100000 

51234.25 

-1233.75 

C15 

1.72 

C24 

69.2 

75 


Rep  14 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004167963 

-9.63409E-07 

2 

0.007083 

0.002083 

0.006978624 

0.000104376 

3 

0.008333 

0.0025 

0.008754107 

-0.000421107 

4 

0.010417 

0.002917 

0.010693731 

-0.000276731 

5 

0.011666667 

0.002916667 

0.011857614 

-0.000190947 

6 

0.0125 

0.00375 

0.012049526 

0.000450474 

7 

0.012917 

0.00375 

0.012902973 

1.40266E-05 

8 

0.020833 

0.005833 

0.021691275 

-0.000858275 

9 

0.02125 

0.005833 

0.021673649 

-0.000423649 

10 

0.025 

0.007083 

0.024969314 

3.06864E-05 

11 

0.033333 

0.009583 

0.03332358 

9.4201  IE-06 

12 

0.034167 

0.009583 

0.032143966 

0.002023034 

13 

0.041667 

0.024167 

0.058721281 

-0.017054281 

14 

50000.5 

3000 

49121.4661 

879.0338983 

15 

50000.5 

6000 

48859.73729 

1140.762712 

16 

50000.5 

10000 

46807.01075 

3193.489247 

17 

50000.5 

15000 

50725.63542 

-725.1354167 

18 

50000.5 

50000 

51623.80645 

-1623.306452 

19 

50000.5 

75000 

37507.61538 

12492.88462 

20 

50000.5 

1-100000 

48433.59633 

1566.90367 

C15 

1.8 

C24 

66.27 

76 


Rep  15 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004191591 

-2.45914E-05 

2 

0.007083 

0.002083 

0.007223101 

-0.000140101 

3 

0.008333 

0.0025 

0.008134346 

0.000198654 

4 

0.010417 

0.002917 

0.010977536 

-0.000560536 

5 

0.011666667 

0.002916667 

0.011393247 

0.00027342 

6 

0.0125 

0.00375 

0.01268985 

-0.00018985 

7 

0.012917 

0.00375 

0.011873602 

0.001043398 

8 

0.020833 

0.005833 

0.019745399 

0.001087601 

9 

0.02125 

0.005833 

0.022835427 

-0.001585427 

10 

0.025 

0.007083 

0.025452017 

-0.000452017 

11 

0.033333 

0.009583 

0.03154238 

0.00179062 

12 

0.034167 

0.009583 

0.032833447 

0.001333553 

13 

0.041667 

0.024167 

0.04124246 

0.00042454 

14 

50000.5 

3000 

50636.87963 

-636.3796296 

15 

50000.5 

6000 

47585.40385 

2415.096154 

16 

50000.5 

10000 

49669.78351 

330.7164948 

17 

50000.5 

15000 

46799.95 

3200.55 

18 

50000.5 

50000 

52507.10345 

-2506.603448 

19 

50000.5 

75000 

53399.61538 

-3399.115385 

20 

50000.5 

1-100000 

48735.76585 

1264.734146 

C15 

1.74 

C24 

69.19 

77 


Rep  16 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004169741 

-2.741 14E-06 

2 

0.007083 

0.002083 

0.0068698 

0.0002132 

3 

0.008333 

0.0025 

0.008669155 

-0.000336155 

4 

0.010417 

0.002917 

0.010399832 

1.71677E-05 

5 

0.011666667 

0.002916667 

0.011851221 

-0.000184554 

6 

0.0125 

0.00375 

0.012065369 

0.000434631 

7 

0.012917 

0.00375 

0.013386899 

-0.000469899 

8 

0.020833 

0.005833 

0.021046025 

-0.000213025 

9 

0.02125 

0.005833 

0.022496201 

-0.001246201 

10 

0.025 

0.007083 

0.024827136 

0.000172864 

11 

0.033333 

0.009583 

0.035709933 

-0.002376933 

12 

0.034167 

0.009583 

0.035568933 

-0.001401933 

13 

0.041667 

0.024167 

0.050353959 

-0.008686959 

14 

50000.5 

3000 

53109 

-3108.5 

15 

50000.5 

6000 

48970.35714 

1030.142857 

16 

50000.5 

10000 

46694.4433 

3306.056701 

17 

50000.5 

15000 

50540.28866 

-539.7886598 

18 

50000.5 

50000 

45379.79894 

4620.701058 

19 

50000.5 

75000 

46829.4 

3171.1 

20 

50000.5 

1-100000 

47837.14085 

2163.359155 

C15 

1.78 

C24 

66.1 

78 


Rep  17 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004256841 

-8.9841  IE-05 

2 

0.007083 

0.002083 

0.006739458 

0.000343542 

3 

0.008333 

0.0025 

0.008894049 

-0.000561049 

4 

0.010417 

0.002917 

0.010576386 

-0.000159386 

5 

0.011666667 

0.002916667 

0.011238023 

0.000428644 

6 

0.0125 

0.00375 

0.011712049 

0.000787951 

7 

0.012917 

0.00375 

0.01287488 

4.21202E-05 

8 

0.020833 

0.005833 

0.020781998 

5.10021E-05 

9 

0.02125 

0.005833 

0.020599492 

0.000650508 

10 

0.025 

0.007083 

0.026079718 

-0.001079718 

11 

0.033333 

0.009583 

0.034535543 

-0.001202543 

12 

0.034167 

0.009583 

0.032215424 

0.001951576 

13 

0.041667 

0.024167 

0.046186727 

-0.004519727 

14 

50000.5 

3000 

48274.79825 

1725.701754 

15 

50000.5 

6000 

44195.92793 

5804.572072 

16 

50000.5 

10000 

57200.36471 

-7199.864706 

17 

50000.5 

15000 

47580.43396 

2420.066038 

18 

50000.5 

50000 

48324.37634 

1676.123656 

19 

50000.5 

75000 

58223.875 

-8223.375 

20 

50000.5 

1-100000 

47567.45327 

2433.046729 

C15 

1.76 

C24 

67.91 

79 


Rep  18 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004243814 

-7.68141E-05 

2 

0.007083 

0.002083 

0.006953407 

0.000129593 

3 

0.008333 

0.0025 

0.008295955 

3.70446E-05 

4 

0.010417 

0.002917 

0.010534511 

-0.000117511 

5 

0.011666667 

0.002916667 

0.011194451 

0.000472215 

6 

0.0125 

0.00375 

0.012069798 

0.000430202 

7 

0.012917 

0.00375 

0.013503365 

-0.000586365 

8 

0.020833 

0.005833 

0.020774826 

5.81743E-05 

9 

0.02125 

0.005833 

0.020841265 

0.000408735 

10 

0.025 

0.007083 

0.025610213 

-0.000610213 

11 

0.033333 

0.009583 

0.031770122 

0.001562878 

12 

0.034167 

0.009583 

0.036062114 

-0.001895114 

13 

0.041667 

0.024167 

0.034932977 

0.006734023 

14 

50000.5 

3000 

50649.69 

-649.19 

15 

50000.5 

6000 

46157.04082 

3843.459184 

16 

50000.5 

10000 

49942.375 

58.125 

17 

50000.5 

15000 

59082.76829 

-9082.268293 

18 

50000.5 

50000 

47631.4359 

2369.064103 

19 

50000.5 

75000 

39317.6875 

10682.8125 

20 

50000.5 

1-100000 

51352.35233 

-1351.852332 

C15 

1.58 

C24 

69.26 

80 


Rep  19 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004172572 

-5.57189E-06 

2 

0.007083 

0.002083 

0.007005372 

7.76282E-05 

3 

0.008333 

0.0025 

0.008481414 

-0.000148414 

4 

0.010417 

0.002917 

0.01017309 

0.00024391 

5 

0.011666667 

0.002916667 

0.012579906 

-0.000913239 

6 

0.0125 

0.00375 

0.012068859 

0.000431141 

7 

0.012917 

0.00375 

0.013108031 

-0.000191031 

8 

0.020833 

0.005833 

0.020233855 

0.000599145 

9 

0.02125 

0.005833 

0.020024877 

0.001225123 

10 

0.025 

0.007083 

0.026129789 

-0.001129789 

11 

0.033333 

0.009583 

0.033991102 

-0.000658102 

12 

0.034167 

0.009583 

0.033275221 

0.000891779 

13 

0.041667 

0.024167 

0.031004508 

0.010662492 

14 

50000.5 

3000 

55158.86777 

-5158.367769 

15 

50000.5 

6000 

48190.68033 

1809.819672 

16 

50000.5 

10000 

46790.71875 

3209.78125 

17 

50000.5 

15000 

47517.44954 

2483.050459 

18 

50000.5 

50000 

51620.65445 

-1620.15445 

19 

50000.5 

75000 

51957 

-1956.5 

20 

50000.5 

1-100000 

48992.35593 

1008.144068 

C15 

1.92 

C24 

73.81 

81 


Rep  20 


Control 

Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004138512 

2.84884E-05 

2 

0.007083 

0.002083 

0.00713193 

-4.893E-05 

3 

0.008333 

0.0025 

0.008357 

-2.40004E-05 

4 

0.010417 

0.002917 

0.01021551 

0.00020149 

5 

0.011666667 

0.002916667 

0.011497314 

0.000169352 

6 

0.0125 

0.00375 

0.012875794 

-0.000375794 

7 

0.012917 

0.00375 

0.013229537 

-0.000312537 

8 

0.020833 

0.005833 

0.020663792 

0.000169208 

9 

0.02125 

0.005833 

0.020277746 

0.000972254 

10 

0.025 

0.007083 

0.025336316 

-0.000336316 

11 

0.033333 

0.009583 

0.034095713 

-0.000762713 

12 

0.034167 

0.009583 

0.033189736 

0.000977264 

13 

0.041667 

0.024167 

0.033001666 

0.008665334 

14 

50000.5 

3000 

47415.07258 

2585.427419 

15 

50000.5 

6000 

51627.89744 

-1627.397436 

16 

50000.5 

10000 

48821.83168 

1178.668317 

17 

50000.5 

15000 

52430.9646 

-2430.464602 

18 

50000.5 

50000 

49870.64737 

129.8526316 

19 

50000.5 

75000 

54812.33333 

-4811.833333 

20 

50000.5 

1-100000 

49447.28139 

553.2186147 

C15 

1.9 

C24 

65.04 

82 


Rep  21 


Control  Parameter  1 

Parameter2 

Value 

difference 

1 

0.004167 

0.00125 

0.004310305 

-0.000143305 

2 

0.007083 

0.002083 

0.00657351 

0.00050949 

3 

0.008333 

0.0025 

0.008162304 

0.000170696 

4 

0.010417 

0.002917 

0.010277619 

0.000139381 

5 

0.011666667 

0.002916667 

0.011450791 

0.000215875 

6 

0.0125 

0.00375 

0.012367589 

0.000132411 

7 

0.012917 

0.00375 

0.011417388 

0.001499612 

8 

0.020833 

0.005833 

0.020482607 

0.000350393 

9 

0.02125 

0.005833 

0.019570145 

0.001679855 

10 

0.025 

0.007083 

0.025038767 

-3.87669E-05 

11 

0.033333 

0.009583 

0.032080453 

0.001252547 

12 

0.034167 

0.009583 

0.03355041 

0.00061659 

13 

0.041667 

0.024167 

0.035922637 

0.005744363 

14 

50000.5 

3000 

53590.32692 

-3589.826923 

15 

50000.5 

6000 

47324.65347 

2675.846535 

16 

50000.5 

10000 

50483.9186 

-483.4186047 

17 

50000.5 

15000 

52293.42391 

-2292.923913 

18 

50000.5 

50000 

50470.09827 

-469.5982659 

19 

50000.5 

75000 

54178.18182 

-4177.681818 

20 

50000.5 

1-100000 

50583.61386 

-583.1138614 

C15 

1.7 

C24 

69.48 

83 


Appendix  G:  20  X  21  Matrix  of  Potential  Controls  and  Output  Variables 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

C15 

C25 

1 

-IE-04 

2E-04 

IE-04 

-3E-04 

-0.001 

7E-04 

-5E-04 

4E-04 

-0.001 

-8E-04 

-4E-04 

0.002 

0.004 

-2359 

1115 

2398 

1145 

2310 

-10801 

654.2 

1.72 

67.91 

2 

5E-05 

3E-04 

-2E-04 

-5E-04 

5E-04 

IE-04 

-2E-04 

-6E-04 

-0.001 

-3E-04 

4E-05 

-5E-04 

-0.004 

-2279 

1321 

2183 

-3965 

956.6 

6832 

2529 

1.68 

68.76 

3 

-IE-04 

-2E-05 

-6E-04 

-2E-05 

7E-04 

-IE-05 

4E-04 

-2E-04 

4E-04 

-0.002 

-0.002 

0.002 

-0.004 

1999 

2846 

-387.3 

2464 

1172 

20016 

503 

1.72 

68.14 

4 

2E-04 

-2E-04 

-5E-05 

IE-04 

-4E-04 

-3E-04 

-7E-04 

-0.001 

7E-04 

-0.001 

0.003 

5E-04 

-0.007 

3256 

-3270 

-3715 

-2111 

-603.2 

26 

902.4 

1.56 

69.53 

5 

2E-05 

-2E-05 

IE-04 

-6E-05 

8E-04 

-3E-04 

3E-04 

-3E-04 

-3E-04 

-2E-04 

-0.002 

-5E-06 

-0.009 

3750 

-2246 

-1623 

169.1 

-134 

15635 

-2180 

1.48 

70.55 

6 

4E-05 

-2E-04 

2E-04 

5E-04 

-4E-04 

-3E-05 

-7E-04 

-9E-04 

3E-05 

-2E-04 

-0.003 

-4E-04 

3E-04 

-3744 

-69.94 

6246 

2326 

1638 

-6075 

795 

1.84 

67.34 

7 

-4E-05 

-3E-04 

2E-04 

IE-04 

-7E-04 

-IE-06 

-3E-04 

-7E-04 

0.001 

-0.002 

0.002 

0.004 

-0.005 

-2818 

-2280 

2700 

-819.8 

-2517 

-15022 

-5130 

1.74 

66.6 

8 

-3E-05 

-3E-04 

-2E-04 

-2E-05 

2E-04 

-2E-04 

-4E-04 

-5E-05 

0.001 

3E-04 

3E-05 

-4E-04 

-0.007 

758.7 

-4307 

-4024 

-4001 

-490.8 

775 

1489 

1.7 

68.29 

9 

-3E-07 

2E-04 

IE-04 

-0.001 

3E-04 

7E-04 

5E-04 

0.001 

-0.002 

7E-05 

5E-04 

-0.002 

-9E-04 

-1542 

-2593 

0.125 

-312.6 

2258 

-1739 

2845 

1.62 

68.86 

10 

6E-05 

-IE-04 

-IE-04 

-4E-05 

-3E-04 

-9E-05 

IE-03 

-IE-04 

-0.003 

-5E-04 

0.005 

5E-04 

0.003 

-1209 

276.8 

-5195 

3565 

2630 

-11637 

-1692 

1.84 

66.97 

11 

5E-05 

3E-05 

-4E-04 

-2E-04 

-6E-06 

-5E-04 

2E-04 

-0.001 

-0.002 

-0.002 

0.001 

-0.001 

-0.015 

2827 

1570 

252.7 

6018 

-893.2 

-2407 

4995 

1.76 

66.78 

12 

IE-04 

2E-04 

2E-04 

-9E-06 

-IE-04 

-0.001 

-4E-04 

-9E-04 

-3E-04 

-IE-04 

-0.002 

-0.002 

-0.007 

-306 

-2495 

525.3 

4945 

-1336 

-25419 

1993 

1.6 

68.97 

13 

-3E-05 

-6E-05 

2E-04 

6E-04 

4E-04 

0.001 

4E-04 

5E-04 

-6E-04 

2E-04 

-0.002 

8E-04 

0.006 

-3352 

-89.86 

4028 

1338 

2224 

-11490 

-1234 

1.72 

69.2 

14 

-IE-06 

IE-04 

-4E-04 

-3E-04 

-2E-04 

5E-04 

IE-05 

-9E-04 

-4E-04 

3E-05 

9E-06 

0.002 

-0.017 

879 

1141 

3193 

-725.1 

-1623 

12493 

1567 

1.8 

66.27 

15 

-2E-05 

-IE-04 

2E-04 

-6E-04 

3E-04 

-2E-04 

0.001 

0.001 

-0.002 

-5E-04 

0.002 

0.001 

4E-04 

-636.4 

2415 

330.7 

3201 

-2507 

-3399 

1265 

1.74 

69.19 

16 

-3E-06 

2E-04 

-3E-04 

2E-05 

-2E-04 

4E-04 

-5E-04 

-2E-04 

-0.001 

2E-04 

-0.002 

-0.001 

-0.009 

-3109 

1030 

3306 

-539.8 

4621 

3171 

2163 

1.78 

66.1 

17 

-9E-05 

3E-04 

-6E-04 

-2E-04 

4E-04 

8E-04 

4E-05 

5E-05 

7E-04 

-0.001 

-0.001 

0.002 

-0.005 

1726 

5805 

-7200 

2420 

1676 

-8223 

2433 

1.76 

67.91 

18 

-8E-05 

IE-04 

4E-05 

-IE-04 

5E-04 

4E-04 

-6E-04 

6E-05 

4E-04 

-6E-04 

0.002 

-0.002 

0.007 

-649.2 

3843 

58.13 

-9082 

2369 

10683 

-1352 

1.58 

69.26 

19 

-6E-06 

8E-05 

-IE-04 

2E-04 

-9E-04 

4E-04 

-2E-04 

6E-04 

0.001 

-0.001 

-7E-04 

9E-04 

0.011 

-5158 

1810 

3210 

2483 

-1620 

-1957 

1008 

1.92 

73.81 

20 

3E-05 

-5E-05 

-2E-05 

2E-04 

2E-04 

-4E-04 

-3E-04 

2E-04 

IE-03 

-3E-04 

-8E-04 

IE-03 

0.009 

2585 

-1627 

1179 

-2430 

129.9 

-4812 

553.2 

1.9 

65.04 

21 

-IE-04 

5E-04 

2E-04 

IE-04 

2E-04 

IE-04 

0.001 

4E-04 

0.002 

-4E-05 

0.001 

6E-04 

0.006 

-3590 

2676 

-483.4 

-2293 

-469.6 

-4178 

-583.1 

1.7 

69.48 
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Appendix  H:  Results  for  Common  Random  Numbers 


Base  ~C  15 


Replication 

unconstrained 

constrained 

difference 

1 

2.02 

1.95 

0.07 

2 

2.16 

1.98 

0.18 

3 

2.26 

1.92 

0.34 

4 

2.15 

2 

0.15 

5 

2.25 

2.02 

0.23 

6 

2.25 

1.93 

0.32 

7 

2.33 

1.99 

0.34 

8 

2.13 

1.98 

0.15 

9 

2.11 

1.94 

0.17 

10 

2.33 

2.04 

0.29 

11 

2.12 

1.98 

0.14 

12 

2.14 

1.88 

0.26 

13 

2.15 

1.99 

0.16 

14 

2.22 

1.88 

0.34 

15 

2.18 

1.92 

0.26 

16 

2.18 

1.95 

0.23 

17 

2.27 

2.04 

0.23 

18 

2.4 

2.04 

0.36 

19 

2.17 

1.94 

0.23 

20 

2.13 

1.9 

0.23 

21 

2.2 

1.99 

0.21 

22 

2.17 

1.95 

0.22 

23 

2.25 

2 

0.25 

24 

2.17 

1.86 

0.31 

25 

2.15 

1.94 

0.21 

26 

2.29 

2.01 

0.28 

27 

2.25 

1.96 

0.29 

28 

2.2 

1.95 

0.25 

29 

2.27 

1.99 

0.28 

30 

2.18 

1.97 

0.21 

Mean 

2.202666667 

1.963 

0.23966667 

St  Dev 

0.07851788 

0.04742689 

0.06910678 

t 

19.3200938 

95%  Cl: 

0.215352775 

0.26398056 

Cl  halfwidth: 

0.024313892 

CRN3  —  C15 


Replication 

unconstrained 

constrained 

difference 

1 

2.18 

1.95 

0.23 

2 

2.13 

1.94 

0.19 

3 

2.03 

2 

0.03 

4 

2.15 

1.9 

0.25 

5 

2.27 

1.94 

0.33 

6 

2.24 

1.95 

0.29 

7 

2.12 

2.02 

0.1 

8 

2.13 

1.97 

0.16 

9 

2.2 

1.99 

0.21 

10 

2.23 

1.93 

0.3 

11 

2.17 

2.01 

0.16 

12 

2.25 

1.97 

0.28 

13 

2.16 

1.94 

0.22 

14 

2.18 

1.87 

0.31 

15 

1.97 

1.92 

0.05 

16 

2.17 

2.02 

0.15 

17 

2.07 

2.01 

0.06 

18 

2.2 

1.94 

0.26 

19 

2.05 

1.92 

0.13 

20 

2 

1.96 

0.04 

21 

2.14 

1.98 

0.16 

22 

2.19 

1.94 

0.25 

23 

2.15 

1.99 

0.16 

24 

2.37 

1.93 

0.44 

25 

2.2 

1.97 

0.23 

26 

2.15 

1.92 

0.23 

27 

2.06 

1.95 

0.11 

28 

2.27 

2.03 

0.24 

29 

2.15 

1.97 

0.18 

30 

2.2 

1.95 

0.25 

Mean 

2.159333333 

1.95933333 

0.2 

St  Dev 

0.084890734 

0.03805018 

0.09395524 

t 

11.8585412 

95%  Cl: 

0.166943658 

0.23305634 

Cl  halfwidth: 

0.033056342 

Improvement: 

-35.96% 
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Base  —  C24 


Replication 

unconstrained 

constrained 

difference 

1 

68.26 

61.74 

6.52 

2 

68.81 

61.66 

7.15 

3 

66.37 

65.16 

1.21 

4 

67.19 

61.81 

5.38 

5 

67.01 

61.61 

5.4 

6 

66.74 

65.04 

1.7 

7 

65.29 

60.77 

4.52 

8 

68.62 

61.37 

7.25 

9 

67.35 

61.74 

5.61 

10 

65.71 

62.8 

2.91 

11 

67.28 

62.33 

4.95 

12 

66.96 

65.11 

1.85 

13 

67.35 

63.11 

4.24 

14 

65.72 

64.07 

1.65 

15 

67.61 

65.92 

1.69 

16 

67.17 

61.87 

5.3 

17 

66.29 

62.6 

3.69 

18 

64.67 

62.8 

1.87 

19 

66.54 

65.45 

1.09 

20 

67.26 

59.93 

7.33 

21 

66.95 

60.74 

6.21 

22 

67.54 

61.44 

6.1 

23 

66.85 

61.86 

4.99 

24 

67.28 

64.29 

2.99 

25 

67.18 

63.03 

4.15 

26 

66.43 

61.74 

4.69 

27 

66.97 

61.8 

5.17 

28 

66.4 

65.3 

1.1 

29 

66.6 

62.45 

4.15 

30 

66.79 

65.74 

1.05 

Mean 

66.90633333 

62.8426667 

4.06366667 

St  Dev 

0.875090077 

1.68354702 

2.03690855 

t 

11.1139593 

95%  Cl: 

3.347019569 

4.78031376 

Cl  halfwidth: 

0.716647098 

CRN3  -  C24 


Replication 

unconstrained 

constrained 

difference 

1 

67.28 

61.29 

5.99 

2 

68.18 

65.17 

3.01 

3 

68.84 

61.69 

7.15 

4 

67.22 

64.66 

2.56 

5 

66.73 

65.26 

1.47 

6 

66.93 

60.66 

6.27 

7 

67 

61.25 

5.75 

8 

67.13 

61.54 

5.59 

9 

67.65 

62.72 

4.93 

10 

67.24 

61.86 

5.38 

11 

67.09 

62.24 

4.85 

12 

66.29 

61.33 

4.96 

13 

67.08 

64.72 

2.36 

14 

67.19 

65.08 

2.11 

15 

69.28 

65.33 

3.95 

16 

67.49 

61.87 

5.62 

17 

67.52 

61.66 

5.86 

18 

66.93 

61.25 

5.68 

19 

68.81 

64.98 

3.83 

20 

68.8 

61.34 

7.46 

21 

67.5 

62.81 

4.69 

22 

66.9 

65.65 

1.25 

23 

68.03 

62.77 

5.26 

24 

65.34 

64.9 

0.44 

25 

67.43 

62.07 

5.36 

26 

67.2 

61.13 

6.07 

27 

68.89 

62.19 

6.7 

28 

65.86 

62.89 

2.97 

29 

67.3 

65.69 

1.61 

30 

67.28 

61.7 

5.58 

Mean 

67.4136667 

62.9233333 

4.49033333 

St  Dev 

0.88028398 

1.68821949 

1.88867858 

t 

13.24472 

95%  Cl: 

3.8258381 

5.15482857 

Cl  halfwidth: 

0.66449523 

Improvement: 

7.28% 
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Appendix  I:  Results  for  Antithetic  Variates 


Replication 

Base  -- 
C15 

AV  —  C15 

1 

2.02 

2.02 

2 

2.16 

2.16 

3 

2.26 

2.26 

4 

2.15 

2.15 

5 

2.25 

2.25 

6 

2.25 

2.25 

7 

2.33 

2.33 

8 

2.13 

2.13 

9 

2.11 

2.11 

10 

2.33 

2.33 

11 

2.12 

2.12 

12 

2.14 

2.14 

13 

2.15 

2.15 

14 

2.22 

2.22 

15 

2.18 

2.18 

16 

2.18 

2.04 

17 

2.27 

2.15 

18 

2.4 

2.26 

19 

2.17 

2.02 

20 

2.13 

2.27 

21 

2.2 

2.02 

22 

2.17 

2.19 

23 

2.25 

2.2 

24 

2.17 

2.05 

25 

2.15 

2.1 

26 

2.29 

2.27 

27 

2.25 

2.18 

28 

2.2 

2.34 

29 

2.27 

2.37 

30 

2.18 

2.17 

Mean 

2.20266667 

2.181 

St  Dev 

0.07851788 

0.098797948 

95%  Cl,  base: 

halfwidth: 

0.02809675 

2.17456992 

2.23076341 

95%  Cl,  AV: 

halfwidth: 

0.03535374 

2.145646258 

2.21635374 

Improvement: 

-25.83% 

Replication 

Base  — 
C24 

AV  -  C24 

1 

68.26 

68.26 

2 

68.81 

68.81 

3 

66.37 

66.37 

4 

67.19 

67.19 

5 

67.01 

67.01 

6 

66.74 

66.74 

7 

65.29 

65.29 

8 

68.62 

68.62 

9 

67.35 

67.35 

10 

65.71 

65.71 

11 

67.28 

67.28 

12 

66.96 

66.96 

13 

67.35 

67.35 

14 

65.72 

65.72 

15 

67.61 

67.61 

16 

67.17 

68.87 

17 

66.29 

67.07 

18 

64.67 

66.24 

19 

66.54 

69.14 

20 

67.26 

66.62 

21 

66.95 

69.01 

22 

67.54 

67.3 

23 

66.85 

67.55 

24 

67.28 

68.9 

25 

67.18 

67.41 

26 

66.43 

65.48 

27 

66.97 

67.19 

28 

66.4 

65.94 

29 

66.6 

64.87 

30 

66.79 

67.63 

Mean 

66.9063333 

67.183 

St  Dev 

0.87509008 

1.164501225 

95%  Cl,  base: 

halfwidth: 

0.31314121 

66.59319212 

67.2194745 

95%  Cl,  AV: 

halfwidth: 

0.41670375 

66.76629625 

67.5997038 

Improvement: 

-33.07% 
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Appendix  J:  Results  for  Control  Variates 


C15  -  Single  Control: 

x  -  mu  X  Y  sq  5  Y  -  Y(bar)  product5  Yb  Yhat  b  Y-YharbA2 


5  5  C 1 5  x  -  xbar5 


-0.0013 

0.012967 

1.72 

0.001297352 

1.68312E-06 

-0.001905 

-2.471 15E-06 

1.730605 

1.615466376 

0.010927279 

0.000495 

0.011172 

1.68 

-0.000497929 

2.47934E-07 

-0.041905 

2.08656E-05 

1.683537 

1.772232628 

0.008506858 

0.000663 

0.011004 

1.72 

-0.000665954 

4.43495E-07 

-0.001905 

1.26848E-06 

1.743014 

1.786904797 

0.004476252 

-0.00037 

0.012034 

1.56 

0.000363978 

1.3248E-07 

-0.161905 

-5.89298E-05 

1.577601 

1.696969742 

0.01876071 

0.000837 

0.010829 

1.48 

-0.000840342 

7.06175E-07 

-0.241905 

0.000203283 

1.483159 

1.802132588 

0.103769404 

-0.00044 

0.012106 

1.84 

0.000436685 

1.90693E-07 

0.118095 

5.15704E-05 

1.842519 

1.690620935 

0.022314105 

-0.00074 

0.012403 

1.74 

0.000733203 

5.37587E-07 

0.018095 

1.32675E-05 

1.762675 

1.664728542 

0.005665792 

0.000177 

0.011489 

1.7 

-0.000180584 

3.26105E-08 

-0.021905 

3.95565E-06 

1.696559 

1.744521614 

0.001982174 

0.000347 

0.01132 

1.62 

-0.000349884 

1.22419E-07 

-0.101905 

3.56548E-05 

1.619115 

1.75930512 

0.019405916 

-0.00029 

0.011959 

1.84 

0.000289259 

8.36708E-08 

0.118095 

3.41601E-05 

1.846852 

1.703494327 

0.018633799 

-6.3E-06 

0.011673 

1.76 

3.25838E-06 
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0.038095 

1.24129E-07 

1.779599 

1.728468276 

0.00099425 

-0.00015 

0.011815 

1.6 

0.000145424 

2.1 1482E-08 

-0.121905 

-1.77279E-05 

1.601856 

1.716054178 

0.013468572 

0.000436 

0.011231 

1.72 

-0.000438992 

1.92714E-07 

-0.001905 

8.36175E-07 

1.717451 

1.767086137 

0.002217104 

-0.00019 

0.011858 

1.8 

0.000187859 

3.5291E-08 

0.078095 

1.46709E-05 

1.7996 

1.712348721 

0.007682747 

0.000273 

0.011393 

1.74 

-0.000276508 

7.64568E-08 

0.018095 

-5.00348E-06 

1.745898 

1.75289785 

0.000166355 

-0.00018 

0.011851 

1.78 

0.000181466 

3.29298E-08 

0.058095 

1.05423E-05 

1.777745 

1.712906995 

0.004501471 

0.000429 

0.011238 

1.76 

-0.000431733 

1.86393E-07 

0.038095 

-1.6447E-05 

1.774088 

1.766452248 

4.16315E-05 

0.000472 

0.011194 

1.58 

-0.000475304 

2.25914E-07 

-0.141905 

6.74479E-05 

1.587962 

1.770256931 

0.0361977 

-0.00091 

0.01258 

1.92 

0.000910151 

8.28374E-07 

0.198095 

0.000180296 

1.934741 

1.64927728 

0.073290791 

0.000169 

0.011497 

1.9 

-0.000172441 

2.97358E-08 

0.178095 

-3.07109E-05 

1.904388 

1.74381055 

0.024395144 

0.000216 

0.011451 

1.7 

-0.000218964 

4.79452E-08 

-0.021905 

4.79635E-06 

1.700506 

1.747873013 

0.002291825 

X(bar)  0.01167  1.721905  mu  sub  y  1.729022  sigmaA2  0.025312659 

st  dev  0.047255  error  term  0.034718956 

t  2.131449536 


Dt*D  Dt*DA-l 

0.0476  0.5273 

0.5273  170733 


21 

-6E-05 

-6E-05 

6E-06 
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Y  -  Y (bar) 


product5 


Yb 


Yhat  b 


Y -YharbA2 


C24  -  Single  Control: 

x  -  mu  X  Y  sq  10 

10  10  C25  x  -  xbarlO 


-0.00081 

0.025813 

67.91 

0.000267298 

7.14481E-08 

-0.421429 

-0.000112647 

67.92060537 

68.3279409 

0.17467462 

-0.00027 

0.025271 

68.76 

-0.000274427 

7.531E-08 

0.428571 

-0.000117611 

68.76353707 

68.3350092 

0.18061715 

-0.00176 

0.026764 

68.14 

0.001218348 

1.48437E-06 

-0.191429 

-0.000233227 

68.16301447 

68.3155318 

0.03081142 

-0.00135 

0.026349 

69.53 

0.00080348 

6.4558E-07 

1.198571 

0.000963028 

69.54760135 

68.3209449 

1.46181413 

-0.00024 

0.025242 

70.55 

-0.000303385 

9.20425E-08 

2.218571 

-0.000673081 

70.55315922 

68.3353871 

4.90451042 

-0.00019 

0.025193 

67.34 

-0.00035244 

1.24214E-07 

-0.991429 

0.000349419 

67.34251916 

68.3360271 

0.99207005 

-0.00174 

0.026738 

66.6 

0.001192366 

1.42174E-06 

-1.731429 

-0.002064496 

66.62267545 

68.3158708 

2.94421277 

0.000264 

0.024736 

68.29 

-0.000809216 

6.5483  IE-07 

-0.041429 

3.35247E-05 

68.28655925 

68.341987 

0.00270265 

6.78E-05 

0.024932 

68.86 

-0.000613323 

3.76165E-07 

0.528571 

-0.000324185 

68.85911523 

68.3394311 

0.27099201 

-0.00053 

0.025525 

66.97 

-2.03383E-05 

4.13645E-10 

-1.361429 

2.76891E-05 

66.97685236 

68.3316939 

1.85421039 

-0.0015 

0.026502 

66.78 

0.000956558 

9.15004E-07 

-1.551429 

-0.001484032 

66.79959869 

68.3189476 

2.36835975 

-0.00014 

0.025142 

68.97 

-0.00040324 

1.62603E-07 

0.638571 

-0.000257498 

68.97185633 

68.33669 

0.4010816 

0.000195 

0.024805 

69.2 

-0.000740852 

5.48862E-07 

0.868571 

-0.000643483 

69.19745125 

68.341095 

0.73771772 

3.07E-05 

0.024969 

66.27 

-0.000576199 

3.32005E-07 

-2.061429 

0.001187793 

66.26959961 

68.3389467 

4.28054039 

-0.00045 

0.025452 

69.19 

-9.3495  IE-05 

8.74132E-09 

0.858571 

-8.02722E-05 

69.19589782 

68.3326485 

0.73505164 

0.000173 

0.024827 

66.1 

-0.000718376 

5.16064E-07 

-2.231429 

0.001603005 

66.09774451 

68.3408018 

5.02119263 

-0.00108 

0.02608 

67.91 

0.000534206 

2.85376E-07 

-0.421429 

-0.00022513 

67.92408793 

68.3244584 

0.17177574 

-0.00061 

0.02561 

69.26 

6.47007E-05 

4.18618E-09 

0.928571 

6.00792E-05 

69.26796193 

68.3305844 

0.86381341 

-0.00113 

0.02613 

73.81 

0.000584277 

3.4138E-07 

5.478571 

0.003201003 

73.82474124 

68.3238051 

30.098335 

-0.00034 

0.025336 

65.04 

-0.000209197 

4.37632E-08 

-3.291429 

0.000688555 

65.04438818 

68.3341581 

10.8514777 

-3.9E-05 

0.025039 

69.48 

-0.000506746 

2.56791E-07 

1.148571 

-0.000582033 

69.48050582 

68.3380405 

1.30407156 

X(bar)  0.025546  68.33143 

1.865667 


mu  sub  y  68.3385463 

st  dev  0.008436218 


sigmaA2  4.64333552 

error  term  0.47023268 

t  2.13144954 


Dt*D  Dt*DA-l 

.047620676  0.527295678 

.527295678  170732.9937 


21 


-6.48569E-05 


-6.48569E-05 


5.8573E-06 
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C15  -  Multiple  Controls: 


X  -  mu(x)  Y 


5 

10 

14 

15 

20 

C15 

-0.00130044 

-0.00081281 

-2359.29464 

1115.281818 

654.2028302 

1.72 

0.000494841 

-0.00027109 

-2278.68584 

1320.86036 

2528.726601 

1.68 

0.000662866 

-0.00176386 

1999.417431 

2846.088785 

503.0399061 

1.72 

-0.00036707 

-0.00134899 

3256.365979 

-3269.91414 

902.4289617 

1.56 

0.000837254 

-0.00024213 

3749.912371 

-2245.53125 

-2180.03591 

1.48 

-0.00043977 

-0.00019307 

-3744.31739 

-69.9414414 

794.9794521 

1.84 

-0.00073629 

-0.00173788 

-2818.19492 

-2280.23214 

-5130.31991 

1.74 

0.000177495 

0.000263704 

758.7018349 

-4307.27982 

1488.674757 

1.7 

0.000346796 

6.78101E-05 

-1541.69048 

-2593.08163 

2844.982412 

1.62 

-0.00029235 

-0.00052517 

-1209.33186 

276.8181818 

-1692.35648 

1.84 

-6.3468E-06 

-0.00150207 

2827.261062 

1569.96789 

4994.565421 

1.76 

-0.00014851 

-0.00014227 

-306.020408 

-2494.5 

1992.949198 

1.6 

0.000435903 

0.00019534 

-3351.53571 

-89.864486 

-1233.75 

1.72 

-0.00019095 

3.06864E-05 

879.0338983 

1140.762712 

1566.90367 

1.8 

0.00027342 

-0.00045202 

-636.37963 

2415.096154 

1264.734146 

1.74 

-0.00018455 

0.000172864 

-3108.5 

1030.142857 

2163.359155 

1.78 

0.000428644 

-0.00107972 

1725.701754 

5804.572072 

2433.046729 

1.76 

0.000472215 

-0.00061021 

-649.19 

3843.459184 

-1351.85233 

1.58 

-0.00091324 

-0.00112979 

-5158.36777 

1809.819672 

1008.144068 

1.92 

0.000169352 

-0.00033632 

2585.427419 

-1627.39744 

553.2186147 

1.9 

0.000215875 

-3.8767E-05 

-3589.82692 

2675.846535 

-583.113861 

1.7 

Ybar  1.7219048 
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C15  -  Multiple  Controls  (cont.) 

Y-Y(bar)  product5  productlO  product!4  product!5  product20  sq  5  sq  10  sq  14  sq  15  sq20 


-0.001905 

-2.47E-06 

-5.09E-07 

-3.317521 

1.501129 

0.019568 

1.683E-06 

7.14481E-08 

3033514.524 

621089.9242 

105.5335545 

-0.041905 

2.09E-05 

1.15E-05 

-69.60758 

41.63955 

78.98196 

2.479E-07 

7.531E-08 

2759219.804 

987382.3112 

3552458.688 

-0.001905 

1.27E-06 

-2.32E-06 

4.984787 

4.797904 

-0.268362 

4.435E-07 

1.48437E-06 

6848758.755 

6344854.958 

19849.98407 

-0.161905 

-5.89E-05 

-0.00013 

627.2129 

-582.3882 

41.85223 

1.325E-07 

6.4558E-07 

15007580.61 

12939152.68 

66821.77649 

-0.241905 

0.000203 

7.34E-05 

1056.521 

-622.3533 

-683.1308 

7.062E-07 

9.20425E-08 

19075128.46 

6618890.679 

7974782.779 

0.118095 

5.16E-05 

-4.16E-05 

369.2509 

46.89924 

-17.83824 

1.907E-07 

1.242 14E-07 

9776387.309 

157712.7719 

22815.97394 

0.018095 

1.33E-05 

2.16E-05 

39.82036 

47.18191 

104.4864 

5.376E-07 

1.421 74E-06 

4842636.043 

6798646.227 

33341960.55 

-0.021905 

3.96E-06 

1.77E-05 

30.14747 

-101.5169 

18.50394 

3.261E-08 

6.54831E-07 

1894195.444 

21478303.36 

713593.9117 

-0.101905 

3.57E-05 

6.25E-05 

-94.16964 

-297.5895 

224.2977 

1.224E-07 

3.76165E-07 

853950.793 

8527981.923 

4844632.259 

0.118095 

3.42E-05 

-2.4E-06 

69.8812 

5.948581 

275.9043 

8.367E-08 

4.13645E-10 

350151.4491 

2537.242706 

5458233.952 

0.038095 

1.24E-07 

3.64E-05 

-131.2326 

-47.34395 

-165.7385 

1.062E-1 1 

9.15004E-07 

11867039.46 

1544498.792 

18928029.63 

-0.121905 

-1.77E-05 

4.92E-05 

37.98254 

-343.9774 

164.4519 

2.1 15E-08 

1.62603E-07 

97079.28468 

7961930.123 

1819853.126 

-0.001905 

8.36E-07 

1.41E-06 

-5.207504 

-0.794388 

-3.576533 

1.927E-07 

5.48862E-07 

7474426.93 

173933.8038 

3525681.723 

0.078095 

1.47E-05 

-4.5E-05 

-116.8797 

-63.53621 

-72.07986 

3.529E-08 

3.32005E-07 

2239900.741 

661901.807 

851880.6209 

0.018095 

-5E-06 

-1.69E-06 

0.339896 

-37.78117 

-11.2336 

7.646E-08 

8.74132E-09 

352.8286457 

4359355.314 

385397.9401 

0.058095 

1.05E-05 

-4.17E-05 

144.7097 

-40.83826 

-88.27161 

3.293E-08 

5.16064E-07 

6204603.255 

494143.799 

2308665.329 

0.038095 

-1.64E-05 

2.04E-05 

-89.26848 

-208.6622 

-68.15683 

1.864E-07 

2.85376E-07 

5491043.879 

30001722.78 

3200939.108 

-0.141905 

6.74E-05 

-9.18E-06 

-4.483354 

498.9755 

-283.211 

2.259E-07 

4.18618E-09 

998.1874018 

12364154.37 

3983146.627 

0.198095 

0.00018 

0.000116 

899.5053 

-293.702 

-72.1491 

8.284E-07 

3.4138E-07 

20618609.2 

2198193.022 

132651.9763 

0.178095 

-3.07E-05 

-3.73E-05 

-570.4432 

348.1026 

16.15524 

2.974E-08 

4.37632E-08 

10259358.36 

3820409.043 

8228.533217 

-0.021905 

4.8E-06 

1.11E-05 

-65.10601 

51.44678 

-26.8781 

4.795E-08 

2.56791E-07 

8834157.278 

5516191.125 

1505636.337 

Beta(hat)5 

Beta(hat)10 

Beta(hat)14 

Beta(hat)15 

Beta(hat)20 

87.32128772 

13.04777865 

1.54923E-05 

-1.1933E-05 

-6.1296E-06 
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C15  -  Multiple  Controls  (cont.) 

X 

5  10  14  15  20  x  -  xbar5  x-xbarlO  x-xbarl4  x-xbarl5  x  -  xbar20 


0.012967107 

0.02581281 

52359.79464 

48885.21818 

49346.29717 

0.0012974 

0.000267 

1741.699 

-788.0926 

-10.27295 

0.011171826 

0.025271086 

52279.18584 

48679.63964 

47471.7734 

-0.000498 

-0.000274 

1661.09 

-993.6711 

-1884.797 

0.011003801 

0.026763861 

48001.08257 

47154.41121 

49497.46009 

-0.000666 

0.001218 

-2617.013 

-2518.9 

140.89 

0.012033733 

0.026348992 

46744.13402 

53270.41414 

49098.07104 

0.000364 

0.000803 

-3873.962 

3597.103 

-258.4991 

0.010829413 

0.025242127 

46250.58763 

52246.03125 

52180.53591 

-0.00084 

-0.000303 

-4367.508 

2572.72 

2823.966 

0.01210644 

0.025193072 

53744.81739 

50070.44144 

49205.52055 

0.0004367 

-0.000352 

3126.721 

397.1307 

-151.0496 

0.012402958 

0.026737878 

52818.69492 

52280.73214 

55130.81991 

0.0007332 

0.001192 

2200.599 

2607.421 

5774.25 

0.011489171 

0.024736296 

49241.79817 

54307.77982 

48511.82524 

-0.000181 

-0.000809 

-1376.298 

4634.469 

-844.7449 

0.011319871 

0.02493219 

51542.19048 

52593.58163 

47155.51759 

-0.00035 

-0.000613 

924.0946 

2920.271 

-2201.053 

0.011959014 

0.025525174 

51209.83186 

49723.68182 

51692.85648 

0.0002893 

-2.03E-05 

591.736 

50.37105 

2336.286 

0.011673014 

0.026502071 

47173.23894 

48430.53211 

45005.93458 

3.258E-06 

0.000957 

-3444.857 

-1242.779 

-4350.636 

0.011815179 

0.025142272 

50306.52041 

52495 

48007.5508 

0.0001454 

-0.000403 

-311.5755 

2821.689 

-1349.019 

0.011230763 

0.02480466 

53352.03571 

50090.36449 

51234.25 

-0.000439 

-0.000741 

2733.94 

417.0537 

1877.68 

0.011857614 

0.024969314 

49121.4661 

48859.73729 

48433.59633 

0.0001879 

-0.000576 

-1496.63 

-813.5735 

-922.9738 

0.011393247 

0.025452017 

50636.87963 

47585.40385 

48735.76585 

-0.000277 

-9.35E-05 

18.78373 

-2087.907 

-620.8043 

0.011851221 

0.024827136 

53109 

48970.35714 

47837.14085 

0.0001815 

-0.000718 

2490.904 

-702.9536 

-1519.429 

0.011238023 

0.026079718 

48274.79825 

44195.92793 

47567.45327 

-0.000432 

0.000534 

-2343.298 

-5477.383 

-1789.117 

0.011194451 

0.025610213 

50649.69 

46157.04082 

51352.35233 

-0.000475 

6.47E-05 

31.5941 

-3516.27 

1995.782 

0.012579906 

0.026129789 

55158.86777 

48190.68033 

48992.35593 

0.0009102 

0.000584 

4540.772 

-1482.63 

-364.2142 

0.011497314 

0.025336316 

47415.07258 

51627.89744 

49447.28139 

-0.000172 

-0.000209 

-3203.023 

1954.587 

90.71126 

0.011450791 

0.025038767 

53590.32692 

47324.65347 

50583.61386 

-0.000219 

-0.000507 

2972.231 

-2348.657 

1227.044 

X(bar)  0.011669755  0.025545512  50618.0959  49673.31077  49356.57012 
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C15  -  Multiple  Controls  (cont.) 


Yb 

5 

10 

14 

15 

20 

avg 

1 

1.833556112 

1.730605367 

1.756550875 

1.733309192 

1.724010003 

1.7556063 

2 

1.636789859 

1.683537067 

1.71530206 

1.695762459 

1.695500088 

1.6853783 

3 

1.66211769 

1.743014466 

1.689024447 

1.75396374 

1.723083434 

1.7142408 

4 

1.592052745 

1.577601355 

1.509551437 

1.520978549 

1.56553153 

1.5531431 

5 

1.4068899 

1.483159225 

1.421905274 

1.453203 

1.466637247 

1.4463589 

6 

1.878401552 

1.842519162 

1.898008047 

1.839165355 

1.844872908 

1.8605934 

7 

1.804293945 

1.762675447 

1.78366029 

1.712788898 

1.708553181 

1.7543944 

8 

1.684500873 

1.696559253 

1.688245972 

1.648599168 

1.709124984 

1.685406 

9 

1.589717368 

1.619115229 

1.643884314 

1.589055515 

1.63743861 

1.6158422 

10 

1.86552816 

1.846852356 

1.858735319 

1.843303404 

1.829626528 

1.8488092 

11 

1.760554211 

1.779598687 

1.716199254 

1.778735178 

1.790614698 

1.7651404 

12 

1.612968309 

1.601856334 

1.604740957 

1.570231937 

1.612215985 

1.6004027 

13 

1.68193635 

1.71745125 

1.77192296 

1.718927604 

1.712437604 

1.7205352 

14 

1.816673766 

1.799599611 

1.786381753 

1.813613268 

1.809604496 

1.8051746 

15 

1.716124637 

1.745897823 

1.749858977 

1.768820499 

1.747752317 

1.7456909 

16 

1.796115493 

1.777744513 

1.82815778 

1.792293188 

1.793260531 

1.7975143 

17 

1.722570239 

1.774087926 

1.73326493 

1.829268738 

1.774913608 

1.7668211 

18 

1.538765557 

1.587961926 

1.590057439 

1.625865839 

1.571713683 

1.5828729 

19 

1.999745207 

1.934741242 

1.999914924 

1.941597445 

1.926179522 

1.9604357 

1.885211937 

1.904388176 

1.859945811 

1.880579487 

1.90339101 

1.8867033 

21 

1.681149475 

1.700505822 

1.755614636 

1.731932158 

1.696425744 

1.7131256 
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C15  -  Multiple  Controls  (cont.) 


Corr.  Term 

5 

10 

14 

15 

20 

sum 

Y  hat  b 

Y  -  YhatbA2 

1 

-0.113556 

-0.010605 

-0.036551 

-0.013309 

-0.00401 

-0.1780315 

1.548834599 

0.029297594 

2 

0.04321 

-0.003537 

-0.035302 

-0.015762 

-0.0155 

-0.0268915 

1.699974614 

0.000398985 

3 

0.057882 

-0.023014 

0.030976 

-0.033964 

-0.003083 

0.0287962 

1.755662369 

0.001271805 

4 

-0.032053 

-0.017601 

0.050449 

0.039021 

-0.005532 

0.0342844 

1.76115053 

0.040461536 

5 

0.07311 

-0.003159 

0.058095 

0.026797 

0.013363 

0.1682054 

1.895071501 

0.172284351 

6 

-0.038402 

-0.002519 

-0.058008 

0.000835 

-0.004873 

-0.102967 

1.623899122 

0.046699589 

7 

-0.064294 

-0.022675 

-0.04366 

0.027211 

0.031447 

-0.0719718 

1.654894385 

0.007242966 

8 

0.015499 

0.003441 

0.011754 

0.051401 

-0.009125 

0.0729698 

1.799835897 

0.009967206 

9 

0.030283 

0.000885 

-0.023884 

0.030944 

-0.017439 

0.020789 

1.74765511 

0.016295827 

10 

-0.025528 

-0.006852 

-0.018735 

-0.003303 

0.010373 

-0.0440458 

1.682820379 

0.024705433 

11 

-0.000554 

-0.019599 

0.043801 

-0.018735 

-0.030615 

-0.025702 

1.701164117 

0.003461661 

12 

-0.012968 

-0.001856 

-0.004741 

0.029768 

-0.012216 

-0.0020135 

1.724852624 

0.015588178 

13 

0.038064 

0.002549 

-0.051923 

0.001072 

0.007562 

-0.0026758 

1.724190379 

1.75593E-05 

14 

-0.016674 

0.0004 

0.013618 

-0.013613 

-0.009604 

-0.0258729 

1.700993253 

0.009802336 

15 

0.023875 

-0.005898 

-0.009859 

-0.02882 

-0.007752 

-0.0284543 

1.698411894 

0.001729571 

16 

-0.016115 

0.002255 

-0.048158 

-0.012293 

-0.013261 

-0.0875715 

1.639294642 

0.019797998 

17 

0.03743 

-0.014088 

0.026735 

-0.069269 

-0.014914 

-0.0341054 

1.692760706 

0.004521123 

18 

0.041234 

-0.007962 

-0.010057 

-0.045866 

0.008286 

-0.0143644 

1.712501703 

0.017556701 

19 

-0.079745 

-0.014741 

-0.079915 

-0.021597 

-0.00618 

-0.2021783 

1.524687806 

0.15627173 

20 

0.014788 

-0.004388 

0.040054 

0.019421 

-0.003391 

0.0664836 

1.793349726 

0.011374281 

21 

0.018851 

-0.000506 

-0.055615 

-0.031932 

0.003574 

-0.0656278 

1.661238312 

0.001502468 

sigmaA2  e 
0.039349927 
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C15  -  Multiple  Controls  (cont.) 

Dtranspose  *D  _ 


21 

-6.4857E-05 

-0.01145576 

-12969.5138 

6870.973874 

13522.52743 

-6.48569E-05 

5.8573E-06 

1.8044E-06 

12.49069538 

3.90099115 

1.984491659 

-0.011455761 

1.8044E-06 

1.46101E-05 

-1.39107021 

-11.9238873 

-3.83830288 

-12969.51382 

12.49069538 

-1.39107021 

145539011.1 

-25796443.9 

13628687.46 

6870.973874 

3.90099115 

-11.9238873 

-25796443.9 

135821094.7 

23172907.7 

13522.52743 

1.984491659 

-3.83830288 

13628687.46 

23172907.7 

101352925.8 

(Dtranspose*D)A-l 

0.150811979 

-108.858875 

143.0284843 

2.90808E-05 

1.70548E-05 

-2.0383E-05 

-108.8588749 

326126.9582 

-148988.546 

-0.04551381 

-0.02815239 

0.015052885 

143.0284843 

-148988.546 

218452.9134 

0.03391064 

0.025793192 

-0.01834986 

2.90808E-05 

-0.04551381 

0.03391064 

1.53636E-08 

6.63318E-09 

-5.2871E-09 

1.70548E-05 

-0.02815239 

0.025793192 

6.63318E-09 

1.15635E-08 

-4.2832E-09 

-2.03831E-05 

0.015052885 

-0.01834986 

-5.2871E-09 

-4.2832E-09 

1.32866E-08 
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